yokidans/EthioMart-Amharic-NER

Amharic NER system for Ethiopian e-commerce Telegram channels # EthioMart Amharic NER Project **Building a FinTech-Ready Vendor Intelligence System** *Fine-tuned NER for Telegram Commerce Data with Micro-Lending Analytics* ## 🔍 Project Overview ### Business Context ```mermaid graph LR A[Telegram Channels] --> B[EthioMart Platform] B --> C[Vendor Analytics] C --> D[Micro-Lending Decisions] ``` ## Core Problem: "Transform messy Telegram posts into a smart FinTech engine that reveals which vendors are best candidates for loans." Key Entities: PRODUCT: የቤት ንጥረ ነገር (Home appliances) PRICE: 2500 ብር (ETB) LOC: መከለር (Mekelle) PHONE: 0911223344 ## 🛠️ Technical Architecture 🎯 Task Breakdown ## Task 1: Data Pipeline ### Elite Implementation Key Features: - Multi-channel async scraping - Image OCR with Tesseract-Amharic - Atomic writes with checksum validation ## Task 2: Annotation Protocol ### CoNLL Format Standard ```bash ሽያጭ B-PRODUCT አለኝ O በመከለር B-LOC 2500 B-PRICE ብር I-PRICE ``` ### Quality Control: - IOB2 validation script - Inter-annotator agreement > 0.85 - Entity consistency checks ## Task 3: Model Fine-Tuning ### Hyperparameters ``` yml training: model: xlm-roberta-large batch_size: 16 grad_accum: 4 lr: 2e-5 epochs: 5 warmup: 0.1 optim: use_sam: true # Sharpness-Aware Minimization scheduler: cosine_with_restarts ``` ## Task 4: Model Comparison ## 🏆 Model Benchmark Results | Model | F1-Score | VRAM | Latency | |------------------|----------|--------|---------| | **XLM-RoBERTa-L** | 92.3 | 24 GB | 58 ms | | **AfroXLMR** | 89.7 | 18 GB | 63 ms | | **mBERT** | 85.2 | 16 GB | 47 ms | **Key Insights**: - 🥇 **XLM-RoBERTa-L** delivers best accuracy (F1 92.3) but requires most VRAM - ⚡ **mBERT** offers fastest inference (47ms) with acceptable accuracy tradeoff - ⚖️ **AfroXLMR** provides balanced memory/performance for mid-range GPUs ## Task 5: Model Interpretability ### SHAP Analysis ```python explainer = shap.Explainer(model) shap_values = explainer([sample_text …