zumi123/amharic-data-extractor-week4

# Amharic Telegram Named Entity Recognition (NER) This project extracts and annotates Amharic Telegram messages for Named Entity Recognition (NER). It includes preprocessing, labeling in CoNLL format, and preparation for model training. --- ## Project Structure ``` amharic-data-extractor-week4/ ├── data/ │ ├── cleaned/ # Preprocessed text and labeled subset │ │ ├── processed_messages.csv # Tokenized and cleaned messages │ │ └── amharic_ner_subset.conll # 50 manually labeled messages in CoNLL format │ ├── labeled/ │ │ └── amharic_ner_subset.conll # 50 manually labeled messages in CoNLL format │ ├── raw/ # Raw scraped Telegram messages ├── notebooks/ ├── scripts/ └── README.md ``` --- ## Tasks Overview ### Task 1 – Preprocess Telegram Messages - Tokenize, normalize, and clean Amharic text - Separate metadata (sender, timestamp, etc.) - Output stored in `data/cleaned/processed_messages.csv` Run: ```bash python scripts/preprocess_text.py data/raw/raw_messages_*.csv ``` --- ### Task 2 – Label 50 Messages (CoNLL Format) - Annotate `Product`, `Price`, and `Location` entities - BIO tagging scheme used: - `B-Product`, `I-Product` - `B-PRICE`, `I-PRICE` - `B-LOC`, `I-LOC` - `O` for non-entities Sample format: ``` ፍሪጅ B-Product ቂቤ I-Product 1000 B-PRICE ብር I-PRICE አዲስ B-LOC አበባ I-LOC ``` Use the template generator: ```bash python scripts/create_conll_template.py data/cleaned/processed_messages.csv --n 50 ``` --- ## Setup ### Install dependencies ```bash pip install -r requirements.txt ``` > Optional: For improved Amharic tokenization, install `amseg`: ```bash pip install amseg ``` --- ## Task 3 – Model Training We fine-tuned transformer-based models for Amharic NER using the labeled CoNLL dataset. ### Models Trained - `xlm-roberta-base` (baseline multilingual model) - `afro-xlmr-base` (optimized for African languages) - `xlm-roberta-base-finetuned-amharic` (already fine-tuned on …