OliyadTeshome/ethioMartNER

EthioMartNER is an AI-powered Amharic Named Entity Recognition (NER) system for extracting products, prices, and locations from Telegram e-commerce messages. It powers a centralized shopping hub by fine-tuning multilingual transformers and scoring vendors for micro-lending insights. # EthioMart NER Pipeline An AI-powered Named Entity Recognition (NER) pipeline for Amharic e-commerce text analysis, designed for EthioMart's micro-lending platform. ## 🎯 Project Overview This project implements a comprehensive AI pipeline that: 1. **Scrapes Ethiopian Telegram e-commerce channels** for real-time data 2. **Processes and cleans Amharic text** with advanced NLP techniques 3. **Fine-tunes transformer models** for Amharic NER tasks 4. **Compares multiple models** to select the best performer 5. **Provides model interpretability** using SHAP and LIME 6. **Generates vendor scorecards** for micro-lending decisions ## 🏗️ Architecture ``` ethioMartNER/ ├── src/ # Source code modules │ ├── data/ # Data ingestion and preprocessing │ ├── ner/ # NER model training and evaluation │ ├── interpretability/ # Model interpretability tools │ └── vendor/ # Vendor analytics and scoring ├── scripts/ # Pipeline execution scripts ├── data/ # Data storage │ ├── raw/ # Raw scraped data │ ├── processed/ # Preprocessed data │ └── labelled/ # CONLL format datasets ├── models/ # Trained models ├── outputs/ # Analytics results and scorecards ├── logs/ # Execution logs ├── notebooks/ # Jupyter notebooks for analysis └── tests/ # Unit tests ``` ## 🚀 Quick Start ### 1. Environment Setup ```bash # Clone the repository git clone github.com cd ethioMartNER # Create virtual environment python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate # Install dependencies pip install -r requirements.txt # Install pre-commit hooks pre-commit install ``` ### 2. Configuration 1. **Set up Telegram API credentials:** ```bash # Copy and edit the environment file cp .env.example .env ``` Add your Telegram API credentials to `.env …