RuthKbde/Building-an-Amharic-E-commerce-Data-Extractor

# Building-an-Amharic-E-commerce-Data-Extractor Building an Amharic E-commerce Data Extractor & FinTech Engine Project Overview This project implements a comprehensive pipeline to extract and process e-commerce data from Amharic Telegram channels. The core objective is to transform unstructured textual information (product details, prices, locations, contact info, delivery fees) into structured data. This structured data is then used to generate a "Lending Score" for various vendors, providing valuable insights for micro-lending initiatives in the Ethiopian e-commerce landscape. The pipeline leverages Natural Language Processing (NLP), specifically Named Entity Recognition (NER) using fine-tuned transformer models, to identify key entities within Amharic text. It also incorporates model interpretability techniques (LIME and SHAP) to understand model predictions and offers a vendor analytics engine to derive actionable financial insights. Key Features Telegram Data Scraper: Asynchronously scrapes messages (text, photos, documents) from specified Telegram channels. Amharic Text Preprocessor: Cleans and normalizes Amharic text, handling unique characters, punctuation, and common text artifacts. CoNLL Data Preparation: Generates and reads data in CoNLL format, a standard for NER annotation, facilitating manual labeling. Named Entity Recognition (NER) Model Training: Fine-tunes multilingual transformer models (e.g., mBERT) to identify specific entities like Product, Price, Location, DELIVERY_FEE, and CONTACT_INFO in Amharic text. Model Evaluation: Evaluates the performance of trained NER models using standard metrics (Precision, Recall, F1-score). Model Interpretability (LIME & SHAP): Provides insights into why the NER model makes certain predictions, enhancing trust and debugging capabilities. FinTech Vendor Analytics Engine: Calculates key performance indicators (e.g., activity rate, average views, unique products, contact info completeness …