nuhaminae/Building-an-Amharic-E-Commerce-Data-Extractor

Extracting messages from Ethiopian Telegram based e-commerce channeles fro NER training. # Building an Amharic E-Commerce Data Extractor This repository provides a comprehensive, reproducible framework for extracting, processing, and annotating e-commerce data in Amharic, with an emphasis on informal marketplaces such as Telegram channels. The pipeline is crafted to empower research and applications in natural language processing (NLP), information extraction, and digital commerce analytics for the Amharic language. --- ## Table of Contents - Project Overview - Features - Directory Structure - Installation - Usage - Data - NER Annotation Format - Testing & CI - Contribution - Project Status --- ## Project Overview E-commerce in Ethiopia is rapidly evolving, with a significant amount of trading activity occurring via informal digital channels such as Telegram groups. Unfortunately, there is a dearth of structured datasets and language resources for Amharic, which impedes the development of robust commerce and NLP solutions. This project tackles that gap by providing: - A robust pipeline to **scrape, preprocess, and structure data** from Amharic Telegram channels. - Tools and annotated corpora for **developing and evaluating Named Entity Recognition (NER) and information extraction models** in Amharic. - Example scripts and Jupyter Notebooks for **exploratory data analysis and downstream NLP applications**. --- ## Features - **Data Extraction:** Scripts for scraping raw messages (text, images, metadata) from Telegram channels, focusing on Amharic commerce content. - **Preprocessing:** Advanced cleaning and structuring of raw chat data, including tokenisation, currency normalisation, and entity pattern extraction. - **NER Annotation:** High-quality CoNLL-format annotation of Amharic e-commerce data for entities such as products, prices, phone numbers, and locations. - **DVC Integration:** Data Version Control (DVC) is used for tracking large, processed datasets and ensuring reproducibility. - **Reproducible Workflows:** Jupyter Notebooks and mo …