mekhlua/amharic-E-commerce-data-extractor

# Amharic E-commerce Data Extractor This repository is meant to support the development of a data ingestion and NER pipeline for Amharic e-commerce Telegram channels. The project is broken into several tasks corresponding to the lab assignment: 1. **Data Ingestion & Preprocessing** 2. **Labeling in CoNLL Format** 3. **Fine-tuning NER Model** 4. **Model Comparison & Selection** 5. **Model Interpretability** 6. **Vendor Scorecard and Analytics** --- ## Project Structure (suggested) ``` . ├── data/ # raw and processed data files │ ├── raw/ # scraped messages │ └── processed/ # cleaned/preprocessed dataset ├── notebooks/ # jupyter notebooks for experiments │ ├── 01-labeling.ipynb │ ├── 02-training.ipynb │ └── 03-analysis.ipynb ├── src/ # Python source code │ ├── telegram_scraper.py # scraping/ingestion code │ ├── preprocessing.py # text preprocessing utilities │ ├── labeling.py # CoNLL conversion tools │ ├── train.py # training script using HuggingFace │ ├── evaluate.py # evaluation/comparison logic │ ├── interpret.py # SHAP/LIME utilities │ └── vendor_score.py # vendor analytics engine ├── requirements.txt # Python dependencies └── README.md ``` ## Quick Start 1. **Set up Python environment** (e.g. `python -m venv .venv` and activate) 2. Install dependencies: ```sh pip install -r requirements.txt ``` 3. Follow the task instructions in respective notebooks or scripts. For example, to start ingestion run: ```sh python -m src.telegram_scraper ``` Raw messages will appear in `data/raw/messages.jsonl` and preprocessed records (with normalized text and tokens) will be written to `data/preprocessed/messages.jsonl`. ## Task Guidance Each task in the user request corresponds to a section below: ### Task 1: Data Ingestion and Preprocessing - Identify at least five Ethiopian e-commerce Telegram channels, e.g. channels selli …