Shegaw-21hub/EthioMart-NER

This is a comprehensive and multi-stage machine learning project focused on Amharic NER for e-commerce Telegram data, with the goal of building EthioMart as a centralized platform # EthioMart: Amharic Named Entity Recognition for Telegram E-Commerce ## 📌 Project Overview EthioMart transforms Ethiopia's decentralized Telegram e-commerce into a unified marketplace by extracting business entities (products, prices, locations) from Amharic messages. This end-to-end solution features: - **Automated data pipeline** from Telegram channels - **Custom Amharic NER dataset** with 50 labeled messages - **State-of-the-art multilingual models** fine-tuned for Amharic - **Vendor analytics engine** for micro-lending decisions ## 🏆 Key Achievements ✅ **Data Pipeline** - Collected 1,000+ messages from 5 Telegram channels - Developed preprocessing for Amharic text normalization ✅ **NER Implementation** - Manually labeled 50 messages (600+ tokens) in CoNLL format - Fine-tuned 3 transformer models (F1 scores 0.83-0.88) ✅ **Advanced Features** - Model interpretability with SHAP/LIME - Vendor scoring system for lending decisions ## 📂 Repository Structure ``` EthioMart-NER/ ├── data/ │ ├── raw/ # JSON/CSV from Telegram │ ├── processed/ # Cleaned messages │ └── labeled/ # amharic_ner.conll │ ├── models/ │ ├── xlm-roberta/ # Best model (F1=0.88) │ ├── distilbert/ # Fastest model │ └── mbert/ # Balanced option │ ├── notebooks/ │ ├── 1_data_collection.ipynb │ ├── 2_data_preprocessing.ipynb │ ├── 3_data_labeling.ipynb # CoNLL creation │ ├── 4_model_training.ipynb # Fine-tuning │ ├── 5_model_comparison.ipynb # Benchmarking │ ├── 6_model_interpretability.ipynb │ └── 7_vendor_scorecard.ipynb │ ├── scripts/ │ ├── telegram_scraper.py # Data collection │ ├── data_preprocessor.py # Cleaning pipeline │ └── vendor_analyzer.py # Lending scores │ ├── .env.example # API configuration ├── requirements.txt # Dependencies └── EthioMart_NER_Report.pdf # 15-page final report ``` ```markdown ## 🛠️ Installation Guide …