KEEMBAR/EthioMart-Amharic-NER

EthioMart-Amharic-NER is an NLP project that uses transformer models to extract product names, prices, and locations from Amharic Telegram e-commerce messages, helping EthioMart build a unified shopping platform and identify top vendors for micro-lending. # EthioMart-Amharic-NER ## Project Overview EthioMart-Amharic-NER is a data science project focused on building a Named Entity Recognition (NER) system for Amharic-language e-commerce data collected from Telegram channels. The goal is to extract key business entities (products, prices, locations, etc.) from unstructured messages to power a centralized e-commerce platform for Ethiopia. ## Business Need With the rise of Telegram-based e-commerce in Ethiopia, vendors and customers face challenges due to decentralized channels. EthioMart aims to consolidate real-time data from multiple Telegram channels, enabling seamless product discovery and analytics. This project supports that vision by providing structured, machine-readable data through NER. ## Folder Structure ``` EthioMart-Amharic-NER/ │ ├── data/ # Raw and processed data (not tracked by git, managed by DVC) │ ├── raw/ # Unprocessed, original data from Telegram │ ├── processed/ # Cleaned, tokenized, and labeled data │ ├── scripts/ # Python scripts for scraping, preprocessing, etc. ├── notebooks/ # Jupyter notebooks for EDA, labeling, and analysis ├── models/ # Saved and fine-tuned model files ├── reports/ # Interim and final PDF reports ├── requirements.txt # Python dependencies ├── .gitignore # Files/folders to ignore in git ├── .dvc/ # DVC configuration and cache ├── data.dvc # DVC tracking file for data/ └── README.md # Project overview and instructions ``` ## Task 1: Data Collection, Preprocessing, and EDA - **Data Ingestion:** Messages are scraped from five major Ethiopian e-commerce Telegram channels using a custom Python script (`scripts/telegram_scraper.py`). - **Preprocessing:** Raw messages are cleaned (removal of URLs, emojis, symbols, etc.), normalized, and tokenized. Processed data is saved as per-message JSON files in `data/processed/text/ /`. - **EDA:** Exp …