Natty4/amharic_ecommerce_data_extractor

# Amharic E-commerce NER & Vendor Scorecard 🇪🇹 Extracting Structured Product Intelligence from Telegram E-commerce Channels **Named Entity Recognition (NER) + Vendor Analytics + Micro-Lending Scorecard** --- ## Project Overview This project builds an NLP pipeline to extract and analyze e-commerce information from **Amharic-language Telegram posts**. The system performs: 📌 **Named Entity Recognition (NER)** to identify key entities: - 🛍️ `PRODUCT` – items being sold - 💰 `PRICE` – numerical price values (e.g., "6500 ብር") - 📍 `LOCATION` – delivery/meeting/store areas - 📞 `CONTACT` – phone numbers - **FinTech Vendor Scorecard** to assess vendor activity, engagement, and pricing and identify top-performing microbusinesses eligible for **micro-lending** based on engagement and product insights. ## Quick Start ### 1. Install Dependencies ```bash pip install -r requirements.txt ```` ### 2. Run the Preprocessing & Labeling Pipeline ```bash python scripts/run_preprocessing.py python scripts/run_ner_labeling.py ``` These scripts clean and tokenize Telegram messages and output weakly labeled data in **CoNLL format** (`labeled_data.conll`). ## Pipeline Components ### Data Preprocessing * Scraped Telegram posts using Telethon * Cleaned noise, removed emojis, normalized text ### Rule-based Weak Labeling Custom heuristics for each entity: | Entity | Strategy | | -------- | ----------------------------------------- | | PRODUCT | Keyword anchor matching (e.g., ማሽን, ጫማ) | | PRICE | Regex on digits + ብር / ዋጋ | | LOCATION | Clue phrases (e.g., ቦታ, አድራሻ ሞል) | | CONTACT | Regex (e.g., 09xx,07xx,251 numbers) | * BIO format applied to support model training (`B-`, `I-`, `O`) ### NER Model Training * Used 🤗 HuggingFace transformers * Fine-tuned multilingual models: * `rasyosef/bert-tiny-amharic` * `Davlan/distilbert-base-multilingual-cased-ner-hrl` * `mbeukman/xlm-roberta-base-finetuned-ner-swahili` * …