restinbark/telegram-ecommerce-data-extractor

NER pipeline for Amharic Telegram-based e-commerce data # 📦 Amharic NER System for Telegram-Based E-commerce (EthioMart Project) A multilingual Named Entity Recognition (NER) system fine-tuned for Amharic Telegram messages — built to extract key e-commerce entities such as **Product**, **Price**, and **Location**. This project supports EthioMart’s goal of centralizing fragmented vendor activity into a unified digital platform. --- ## 🚀 Project Overview **Challenge**: Telegram is the dominant channel for informal e-commerce in Ethiopia. Vendors operate in scattered channels, and customers struggle to search or compare products. **Solution**: We built a custom Amharic NER model that extracts structured data from unstructured Telegram messages using open-source LLMs like `xlm-roberta-base` and `bert-base-multilingual-cased`. --- ## ✅ Tasks Completed ### 🔹 Task 1: Data Ingestion & Preprocessing - Connected to 5+ Telegram vendor channels using Telethon. - Scraped messages and metadata. - Cleaned Amharic text and stored in structured format. **Outputs**: ```bash data/raw/ data/processed/ scripts/auth/ scripts/prep/ 🔹 Task 2: Entity Labeling in CoNLL Format Manually labeled 50+ messages. Developed auto_labeling.py to automate BIO tagging using keyword lists. Saved all labels in data/ethio_ner_labels.conll. Key Tags: B-Product, B-PRICE, B-LOC, I-PRICE, I-LOC, O 🔹 Task 3: Model Fine-Tuning (XLM-Roberta) Used Hugging Face Trainer to fine-tune xlm-roberta-base. Token-label alignment using word_ids(). Trained on GPU with validation evaluation. Results: Metric Score F1 Score 0.82 Precision 0.83 Recall 0.81 Accuracy 0.85 ✅ Saved and pushed model to Hugging Face Hub. 🔹 Task 4: Model Comparison & Selection Trained a second model: bert-base-multilingual-cased (mBERT) Reused dataset and evaluation code for apples-to-apples comparison. Comparison: Model F1 Score Precision Recall Accuracy xlm-roberta-base 0.82 0.83 0.81 0.85 bert-base-multilingual-cased 0.76 0.78 0.74 0.80 ✅ Recommendation: Use XLM-Roberta for deploy …