boaztulu/Building-an-Amharic-E-commerce-Data-Extractor

Telegram channel # 🚀 EthioMart Amharic E-Commerce Data Extractor > **A unified pipeline** to scrape, preprocess, and extract key entities from Ethiopian Telegram-based e-commerce channels using cutting-edge NLP techniques. --- *Figure: High-level architecture showing Telegram ingestion → preprocessing → NER model → downstream analytics.* --- ## 📖 Table of Contents 1. 🎯 Project Overview 2. 🛠️ Features 3. ⚙️ Installation & Setup 4. 📦 Project Structure 5. 🚀 Usage 1. Task 1: Data Ingestion 2. Task 2: CoNLL Labeling 3. Task 3: Fine-Tune NER 6. 📊 Sample Outputs 7. 🔧 Configuration 8. 📈 Performance & Metrics 9. 🤝 Contributing 10. 📜 License --- ## 🎯 Project Overview EthioMart aims to **centralize** all Ethiopian Telegram-based e-commerce channels into one platform. This repo provides: - **Real-time scraping** of product posts (text & images) from Telegram channels - **Amharic text cleaning** & normalization - **CoNLL-format** labeling of key entities (Product, Price, Location) - **Fine-tuning** of a multilingual transformer (XLM-RoBERTa) for Amharic NER - **Model explainability** via SHAP & LIME - **Vendor Scorecard** generation for micro-lending insights --- ## 🛠️ Features - ✅ **Asynchronous Telegram Scraper** - 🧹 **Robust Amharic Text Cleaner** - ✍️ **Manual & Automated CoNLL Labeling** - 🤖 **Transformer-based NER Fine-tuning** - 🔍 **Evaluation Metrics**: Precision, Recall, F1, Accuracy - 📊 **Interpretability**: SHAP & LIME visualizations - 💳 **Vendor Analytics Engine**: Posting frequency, views, lending score --- ## ⚙️ Installation & Setup ```bash # Clone the repo git clone github.com cd EthioMart-Data-Extractor # (Optional) Create & activate venv python3 -m venv .venv && source .venv/bin/activate # Install dependencies pip install -r requirements.txt ├── README.md ├── requirements.txt ├── scripts/ │ ├── scrape_telegram.py # Task 1: Async scraper │ ├── text_cleaner.py # Amharic normalization utilities │ …