ayanasamuel8/amharic-text-extractor

# 📚 Amharic NER Pipeline for EthioMart This repository contains tools and scripts to preprocess Amharic Telegram text data, perform Named Entity Recognition (NER) using a fine-tuned transformer model, and extract vendor analytics for micro-lending evaluation. --- ## ✨ Features - **✅ Text Cleaning & Normalization** - Removes emojis, symbols, links, and redundant punctuation. - Normalizes Amharic-specific characters and spacing. - **🔠 Tokenization** - Uses `XLM-RoBERTa` tokenizer for multilingual support. - Token-level formatting for CoNLL-style entity tagging. - **🏷️ NER Annotation Support** - Manual annotation via CoNLL format. - Supports entity labels: - `B-PRICE`, `I-PRICE` - `B-PRODUCT`, `I-PRODUCT` - `B-LOC`, `I-LOC` - `O` for non-entity tokens - **🧠 Model Fine-tuning** - Fine-tunes `xlm-roberta-base` on Amharic NER data. - Built using Hugging Face Transformers. - **📊 Vendor Analytics & Scorecard** - Extracts product/price entities from vendor posts. - Computes posting frequency, engagement metrics, and Lending Score. --- ## 🗂️ Project Structure ``` yaml / ├── scripts/ │ ├── preprocess.py # Cleaning and tokenization │ ├── ner_utils.py # Entity extraction utilities │ ├── vendor_metrics.py # Scorecard computation │ └── telegram_scraper.py # Telegram data scraping │ ├── data/ │ ├── raw/ # Raw Telegram data │ ├── processed/ # Cleaned + enriched data │ └── labeled_conll/ # CoNLL annotated data │ ├── models/ │ └── trained_models/ # Fine-tuned XLM-RoBERTa model │ ├── notebooks/ │ ├── 01_preprocessing.ipynb │ ├── 03_fine_tuning.ipynb │ ├── 05_lime_and_shap.ipynb │ └── 08_scorecard_analysis.ipynb │ ├── interpretability/ │ └── difficult_cases.md # Model errors & discussion │ ├── outputs/ │ └── enriched_data.csv # Product/price enriched records │ ├── tests/ │ ├── test_preprocess.py │ └── test_dummy.py │ ├── requirements.txt └── README.md ``` --- ## 🚀 Getting Started ### 📦 Installation ```bash git clone github.com cd ethiomar …