Mewael-EME/LLM-powered-Amharic-NER-Vendor-Insights-for-EthioMart

Fine-tuning LLMs for Amharic NER to extract products, prices, and locations from Telegram e-commerce posts. Structured data powers EthioMart’s vendor analytics and supports smart micro-lending decisions in Ethiopia’s digital marketplace. # EthioMart Named Entity Recognition & Vendor Analytics ## Project Overview This project focuses on extracting meaningful vendor-related entities from Ethiopian Telegram channels using state-of-the-art Named Entity Recognition (NER) models. The goal is to develop a robust, scalable, and interpretable system to analyze vendor activity and engagement, enabling EthioMart to offer micro-lending to promising vendors. --- ## Features - **NER Model Fine-Tuning:** Fine-tune transformer-based models (XLM-Roberta, mBERT, DistilBERT) for Amharic and multilingual entity extraction. - **End-to-End Data Pipeline:** Seamlessly ingest Telegram channel data from multiple sources, preprocess Amharic text (normalization, cleaning), and prepare structured datasets for training. - **Model Comparison & Selection:** Evaluate multiple models based on accuracy, inference speed, and robustness; apply interpretability methods to explain model decisions. - **Vendor Scorecard Analytics:** Combine extracted entities with metadata (views, timestamps) to calculate key performance metrics and generate a lending score for each vendor. - **Interpretability Tools:** Use SHAP and LIME to ensure transparency and trust in the model's predictions. - **Modular, Reproducible Codebase:** Clear organization, comprehensive documentation, and easy-to-follow setup instructions. --- ## Repository Structure ├── notebooks/ │ ├── task_1_data_preprocessing.ipynb │ ├── task_4_model_comparison.ipynb │ ├── task_5_model_interpretability.ipynb │ ├── task_6_vendor_scorecard.ipynb │ ├── src/ │ ├── data_loader.py │ ├── data_cleaning.py │ ├── train_utils.py │ ├── ner_model.py │ ├── model_comparison.py │ ├── interpretability.py │ ├── vendor_scorecard.py │ ├── data/ # Raw and processed datasets (ignored in git) ├── ner-model/ # Fine-tuned models (ignored in git) ├── results/ # Model checkpoints, logs (ignored in git) ├── .gitignore ├── requirements.txt ├── train_hf_ner.py # Training script for NER models ├── README.md └ …