remurid/Amharic-E-commerce-Data-Extractor

# Amharic E-commerce Data Extractor (Full Project) This repository contains the complete end-to-end implementation for the Amharic E-commerce Data Extractor project. It details a full pipeline from real-time data ingestion from Telegram to advanced model fine-tuning and the creation of a data-driven FinTech vendor scorecard. --- ## Objective The primary goal is to transform unstructured Amharic text from multiple e-commerce Telegram channels into a structured dataset. This data is then used to train a Named Entity Recognition (NER) model that powers a vendor analytics engine, helping EthioMart identify promising vendors for micro-lending opportunities. --- ## Workflow Overview ### Task 1: Data Ingestion and Preprocessing The initial workflow, implemented in local Python scripts, focuses on collecting and cleaning the data. #### 1. Telegram Scraper Setup - A Python script using the Telethon library connects to the Telegram API. - It scrapes messages, images, and metadata (views, timestamps) from 7 specified e-commerce channels. - Raw data is saved to `data/raw/`. #### 2. Preprocessing - A second script cleans the raw text by removing unwanted symbols and normalizing Amharic characters. - The cleaned data is structured and saved to `data/processed/telegram_data_cleaned.csv`, ready for the next stages. #### 3. How to Run - **Scrape Telegram Data:** ```bash python src/services/telegram_scraper.py ``` - **Preprocess the Data:** ```bash python src/services/preprocess_telegram_data.py ``` --- ### Task 2: Manual NER Labeling in CoNLL Format To create training data for our custom model, a subset of 50 messages was manually labeled. - **Goal:** To annotate Product, Price, and Location entities in Amharic text. - **Format:** The CoNLL format was used, where each token is on a new line followed by its NER tag (e.g., B-Product, I-Product, O). - **Output:** The final labeled data is saved as a plain text file (`data/processed/labeled_conll.txt`), which serves as the g …