OliyadTeshome/ethioMartNER
Domaine:
natural language processingType de record:
software
Créateur:
Oli
Hôte:
EthioMartNER is an AI-powered Amharic Named Entity Recognition (NER) system for extracting products, prices, and locations from Telegram e-commerce messages. It powers a centralized shopping hub by fine-tuning multilingual transformers and scoring vendors for micro-lending insights.
# EthioMart NER Pipeline
An AI-powered Named Entity Recognition (NER) pipeline for Amharic e-commerce text analysis, designed for EthioMart's micro-lending platform.
## 🎯 Project Overview
This project implements a comprehensive AI pipeline that:
1. **Scrapes Ethiopian Telegram e-commerce channels** for real-time data
2. **Processes and cleans Amharic text** with advanced NLP techniques
3. **Fine-tunes transformer models** for Amharic NER tasks
4. **Compares multiple models** to select the best performer
5. **Provides model interpretability** using SHAP and LIME
6. **Generates vendor scorecards** for micro-lending decisions
## 🏗️ Architecture
```
ethioMartNER/
├── src/ # Source code modules
│ ├── data/ # Data ingestion and preprocessing
│ ├── ner/ # NER model training and evaluation
│ ├── interpretability/ # Model interpretability tools
│ └── vendor/ # Vendor analytics and scoring
├── scripts/ # Pipeline execution scripts
├── data/ # Data storage
│ ├── raw/ # Raw scraped data
│ ├── processed/ # Preprocessed data
│ └── labelled/ # CONLL format datasets
├── models/ # Trained models
├── outputs/ # Analytics results and scorecards
├── logs/ # Execution logs
├── notebooks/ # Jupyter notebooks for analysis
└── tests/ # Unit tests
```
## 🚀 Quick Start
### 1. Environment Setup
```bash
# Clone the repository
git clone github.com
cd ethioMartNER
# Create virtual environment
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
# Install dependencies
pip install -r requirements.txt
# Install pre-commit hooks
pre-commit install
```
### 2. Configuration
1. **Set up Telegram API credentials:**
```bash
# Copy and edit the environment file
cp .env.example .env
```
Add your Telegram API credentials to `.env …