Selamawit-Alemu/finsl-ethiomart-amharic-ner

# EthioMart Amharic NER System This project builds a Named Entity Recognition (NER) pipeline for extracting product, price, and location information from Amharic e-commerce messages posted on Telegram channels. --- ## 📌 Project Goals - Scrape real-time messages from Ethiopian e-commerce Telegram channels - Preprocess and normalize Amharic text data - Manually label a subset of data in CoNLL format for NER training - Fine-tune transformer models for Amharic entity extraction --- ## 📁 Folder Structure ├── data/ │ ├── raw/ # Raw input files (e.g., Excel with channels) │ ├── clean/ # Preprocessed & labeled data ├── outputs/ # Model outputs, logs ├── photos/ # Downloaded media (images from Telegram) ├── models/ │ └── ner-distilbert/ │ └── WeightedTokenClassification.py ├── notebooks/ │ ├── 01_preprocessing.ipynb │ ├── FinTech Vendor Scorecard for Micro-Lending.ipynb │ ├── interpreting_distilbert.ipynb │ └── model_evaluation.ipynb ├── reports/ │ ├── task4_model_comparison.md │ └── task5.md ├── scripts/ │ ├── Parse_labeled_conll.py │ ├── auto_label_unlabeled.py │ ├── prepare_for_label_studio.py │ ├── preprocess_telegram_data.py │ ├── real_time_ingest.py │ ├── telegram_scrapper.py │ ├── train_distilbert.py │ └── train_ner_model.py ├── requirements.txt └── README.md --- --- ## ✅ Tasks Completed ### **Task 1: Data Collection** - Scraped 5 active Telegram vendors using `telethon` API. - Saved messages, metadata (timestamps, views), and media paths into CSV. ### **Task 2: Data Annotation** - Manually labeled ~500 messages in **CoNLL** format using `label-studio`. - Focused on 3 entity types: `Product`, `Price`, `Location`. ### **Task 3: NER Model Training** - Preprocessed Amharic Telegram messages. - Fine-tuned multilingual transformers: - ✅ `DistilBERT` (final choice) - `XLM-Roberta` (tested but heavier) - Used `class-weighted loss` to tackle class imbalance (many "O" tokens). - Achieved eval loss ~0.085 with DistilBERT. ### **Task 4: Model Comparison** - Compare …