Nama21yo/amharic_ECommerce_data_extractor

# Amharic NER for Ethiopian E-Commerce Telegram Channels ## Project Overview This project develops a Named Entity Recognition (NER) system to extract entities (Product, Price, Location) from Amharic text in Ethiopian e-commerce Telegram channels. It supports EthioMart's micro-lending initiative by analyzing vendor activity through a Vendor Analytics Engine. The project encompasses data collection, preprocessing, labeling, model fine-tuning, comparison, interpretability, and vendor scoring, completed as of June 24, 2025. ### Objectives 1. **Data Ingestion**: Scrape messages from 10 Telegram channels using `Telethon`. 2. **Data Preprocessing**: Normalize and tokenize Amharic text with the `ethiopic` library. 3. **Data Labeling**: Label 30 messages in CoNLL format for NER. 4. **Model Fine-Tuning**: Fine-tune `xlm-roberta-base` for Amharic NER. 5. **Model Comparison**: Compare `xlm-roberta-base`, `distilbert-base-multilingual-cased`, and `bert-base-multilingual-cased` on F1-score, speed, and robustness. 6. **Model Interpretability**: Use SHAP and LIME to explain NER predictions. 7. **Vendor Analytics**: Calculate vendor metrics (posting frequency, views, prices) and a Lending Score. ## Prerequisites - **Python**: 3.8 or higher - **Telegram API Credentials**: API ID, API Hash, and Phone Number from my.telegram.org - **Google Colab**: Recommended for GPU support during model fine-tuning - **Dependencies**: Listed in `requirements.txt` ## Setup Instructions 1. **Clone the Repository**: ```bash git clone github.com cd amharic-ner-telegram ``` 2. **Install Dependencies**: ```bash pip install -r requirements.txt ``` 3. **Configure Telegram API**: - Create a `.env` file in the project root: ``` TELEGRAM_API_ID=your_api_id TELEGRAM_API_HASH=your_api_hash TELEGRAM_PHONE=your_phone_number ``` 4. **Set Up Directory Structure**: ```bash mkdir -p data media/photos media/documents models results ``` ## Project Structure ``` amh …