ET-SPARK/Amharic-E-commerce-Data-Extractor

Transform messy Telegram posts into a smart FinTech engine that reveals which vendors are the best candidates for a loan. # 📦 Amharic E-commerce Data Extractor ## 🎯 Objective Transform raw Telegram posts into a powerful FinTech engine capable of identifying the most loan-worthy vendors using Amharic-language data. --- ## ✅ Task 1: Data Ingestion & Preprocessing ### 🔍 Methodology - Extracted messages from selected Telegram channels using the **Telethon** library. - Normalized Amharic text by: - Removing unwanted characters and extra spaces. - Standardizing Unicode. - Handled media (photos, documents) by storing references for future downloads. - Saved structured messages into a CSV file: `data/preprocessed_data.csv`. - Managed Telegram API credentials securely via a `.env` file (`API_ID`, `API_HASH`, `PHONE`). --- ## ✅ Task 2: Manual Labeling for NER (Named Entity Recognition) ### ✍️ Description - Sampled 40 preprocessed messages from `preprocessed_data.csv`. - Tokenized each message and labeled entities in **CoNLL format** for NER training. - Labeled entities include: - `B-Product`, `I-Product` — Product names and descriptions. - `B-PRICE`, `I-PRICE` — Price information (e.g., "1000 ብር"). - `B-LOC`, `I-LOC` — Location names (e.g., "አዲስ አበባ"). - `O` — Tokens outside of any entity. ### 📄 Output - Labeled dataset saved at `data/conll_raw_sample.txt`. - Each token is on a new line; messages are separated by a blank line. --- ## ✅ Task 3: Fine-Tuning the NER Model ### ✍️ Description - Loaded manually labeled CoNLL data from `data/conll_raw_sample.txt`. - Used the `rasyosef/bert-tiny-amharic` pretrained model from Hugging Face. - Tokenized and aligned labels with subwords. - Fine-tuned the model for recognizing products, prices, and locations using the Hugging Face Trainer API. - Evaluated performance using standard NER metrics. - Saved the fine-tuned model for inference and deployment. --- ## ✅ Task 4: Model Comparison & Selection ### 🔍 Evaluation - Fine-tuned and compared two models: - `bert-tiny-amharic` - `xlm-roberta-base` - Evaluated using F1-score, precision, and r …