AlexKalll/EthioMart

This project automates the extraction of key business entities (products, prices, locations etc) from Amharic Telegram e-commerce channels. It powers EthioMart's centralized marketplace and identifies top vendors for microloans using AI. It includes data preprocessing, model training, and SHAP/LIME interpretability. # EthioMart | Amharic E-commerce Data Extractor ## 📘 Project Overview EthioMart aims to become the primary hub for Telegram-based e-commerce in Ethiopia by centralizing real-time data from various channels. This project focuses on building an Amharic Named Entity Recognition (NER) system to extract key business entities (product names, prices, locations, contacts) from Telegram messages and images. The extracted data will populate EthioMart's centralized database, enabling a seamless experience for customers and informing FinTech initiatives like vendor loan assessments. This project leverages Python, Telegram API, and data science tools to build a robust data pipeline, from scraping and preprocessing to model fine-tuning and interpretability. ## 🏗️ 1. Project Setup, Data Collection and Preprocessing **Deliverables:** * GitHub code for Task 1 (data ingestion and preprocessing). * Data summary (1-2 pages) covering data preparation and labeling steps. ### Repo/Project Structure ```bash EthioMart/ ├── src/ │ ├── telegram_scraper.py # Collects raw data from Telegram channels │ ├── preprocessor.py # Cleans and preprocesses raw text data │ ├── data_labeler.py # Rule-based labeling for NER (Task 2) │ └── model_finetuner.py # Fine-tunes NER models (Task 3 & 4) ├── config/ │ └── config.py # Stores configuration variables (e.g., API credentials, channel list) ├── data/ │ ├── raw/ # Stores raw scraped data (e.g., telegram_data.csv) │ │ └── telegram_data.csv │ ├── processed/ # Stores cleaned and preprocessed data │ │ └── clean_telegram_data.csv │ └── labeled/ # Stores manually and semi-automatically labeled data │ └── telegram_ner_data_rule_based.conll ├── models/ # Stores fine-tuned NER models (Task 3 & 4) │ └── afro_xlmr_ner_fine_tuned/ ├── photos/ # Stores downloaded images from Telegram messages ├── notebooks/ # Jupyter notebooks for EDA …