Abel5173/Amharic-Ecommerce-Data-Extractor

# Amharic E-commerce Data Extractor This project addresses EthioMart’s challenge of consolidating decentralized e-commerce data from Telegram channels in Ethiopia, where vendors post product listings in Amharic and English. The goal is to create a centralized data hub by ingesting, preprocessing, and analyzing Telegram messages, extracting key entities (Product, Price, Location) using a fine-tuned Named Entity Recognition (NER) model, and developing a vendor lending scorecard for data-driven lending decisions. --- ## Project Objectives - **Ingest and preprocess** Amharic and English text and image data from Telegram e-commerce channels. - **Label data** in CoNLL format for NER training. - **Fine-tune an NER model** to extract Product, Price, and Location entities. - **Develop a vendor lending scorecard** (planned) based on posting frequency, engagement, and entity consistency. --- ## Tasks Completed ### Task 1: Data Ingestion and Preprocessing - **Objective:** Collect and preprocess Telegram messages for analysis. - **Scripts:** - `src/core/telegram_scraper.py`: Scrapes messages, images, and metadata using Telethon. - `src/utils/preprocess.py`: Normalizes text, tokenizes Amharic/English, detects language, and extracts emojis. - **Output:** - **Raw data:** `data/raw/telegram_data.csv` (columns: Channel Title, Channel Username, Message ID, Message Text, Date, Media Path). - **Preprocessed data:** `data/processed/preprocessed_telegram_data.csv` (additional columns: Language, Emojis, Preprocessed Text). ### Task 2: CoNLL Labeling - **Objective:** Label messages for NER training in CoNLL format. - **Script:** `src/core/conll_format.py` - Uses regex (e.g., for prices like `2300 ብር`) and a pre-trained NER model (`Davlan/afro-xlmr-mini`) for initial annotations, followed by manual correction. - **Output:** `data/labeled/conll_labeled_data.conll` with 50 messages labeled using BIO tags (`B-PRODUCT`, `I-PRODUCT`, `B-PRICE`, `I-PRICE`, `B-LOC`, `I-LOC`, `O`). ### Ta …