abeladamushumet/Amharic-Ecommerce-Data-Extractor

Modular NLP pipeline that scrapes Amharic Telegram e-commerce posts, fine-tunes multilingual transformers for NER (product, price, location), explains predictions with SHAP/LIME, and produces vendor lending scorecards for micro-finance decisions. # Amharic E-commerce Data Extractor ## Project Overview This project provides a comprehensive solution for extracting, processing, and analyzing e-commerce related data from Telegram channels, specifically focusing on Amharic content. It leverages natural language processing (NLP) techniques, including Named Entity Recognition (NER), to identify key entities like products, prices, and contact information from unstructured text. The extracted data is then cleaned, structured, and made available for further analysis or visualization through a Streamlit dashboard. ## Features - **Telegram Data Scraper**: Extracts messages from specified Telegram channels, focusing on e-commerce related discussions. - **Amharic Text Preprocessing**: Cleans and normalizes Amharic text, including handling of financial entities, contact information, and delivery mentions. - **Named Entity Recognition (NER)**: Fine-tunes a multilingual transformer model (XLM-RoBERTa) to identify and classify e-commerce specific entities within Amharic text. - **Data Interpretability**: Utilizes LIME and SHAP for understanding model predictions and feature importance. - **Interactive Dashboard**: Provides a Streamlit-based dashboard for visualizing extracted data and model insights. - **Vendor Scorecard**: (Implied by directory structure) A component for evaluating vendor performance based on extracted data. ## Installation To set up the project, follow these steps: 1. **Clone the repository**: ```bash git clone github.com cd Amharic-Ecommerce-Data-Extractor ``` 2. **Create a virtual environment** (recommended): ```bash python -m venv venv source venv/bin/activate # On Windows, use `venv\Scripts\activate` ``` 3. **Install dependencies**: ```bash pip install -r requirements.txt ``` The `requirements.txt` file specifies the necessary Python libraries, including `pandas`, `numpy`, `scikit-learn`, `transformers`, `torch`, `seqeval`, `s …