beckhamberhanu/Amharic-NER-Telegram

# EthioMart NER — Amharic Product, Price, and Location Extraction EthioMart NER builds a robust Named Entity Recognition (NER) pipeline for Amharic text to extract business-critical entities — product names, prices, and locations — from Ethiopian e‑commerce Telegram channels. The goal is to power a centralized marketplace by consolidating structured product listings from multiple vendors. --- ## 🚀 Key Features - **Telegram data ingestion**: Collect messages from selected channels using Telethon. - **Amharic-aware preprocessing**: Normalize, clean, and tokenize messages for NER. - **Annotation to CoNLL**: Label and format data for sequence tagging. - **Model fine‑tuning (notebooks)**: Experiment with multilingual transformer models (e.g., XLM‑R, mBERT, AfroXLM‑R). - **Model evaluation & interpretability**: Compare models and inspect decisions. - **Reproducibility**: Version data/process with DVC; containerize with Docker. --- ## 🧰 Tech Stack - **Language**: Python 3.12+ - **Core libraries**: Telethon, Pandas, NumPy, NLTK, tqdm - **Modeling (via notebooks)**: Hugging Face Transformers, Datasets, seqeval, PyTorch - **Tooling**: Jupyter, DVC, Docker > Note: Exact versions are pinned in `requirements.txt`. --- ## 📁 Project Structure ```bash Amharic-NER-Telegram/ ├── data/ │ ├── raw_messages.csv # Raw Telegram messages │ ├── preprocessed_messages.csv # Cleaned/tokenized data │ ├── labeled_data.conll # CoNLL-formatted labels (if available) │ └── labeled_telegram_product_price_location.txt ├── models/ # Trained models and artifacts (if saved) ├── notebooks/ │ ├── Fine_Tune_NER_Model.ipynb │ ├── Label_Dataset_CoNLL.ipynb │ └── Model Comparison, Selection and interpretability.ipynb ├── scripts/ │ ├── config.py # Configuration/utilities │ ├── data_ingestion.py # Fetch messages from Telegram │ └── data_preprocessing.py # Clea …