AlikoGrace/low-resource-ner-twi

Low-resource Named Entity Recognition (NER) for the Twi language: Developing a robust NER system for a low-resource African language with limited labeled data. This project involves data collection (scraped and manually sourced), annotation, model training, and a multilingual baseline approach for African languages # Low-resource Named Entity Recognition (NER) for Twi Language **Project Overview** This project aims to develop a robust Named Entity Recognition (NER) system for the Twi language, a low-resource African language with limited labeled data. By addressing the challenges faced by languages with scarce linguistic resources, this initiative contributes to creating scalable, linguistically-informed NLP models for African languages. The project involves data scraping, manual data collection, annotation, model training, and a multilingual baseline approach for African languages. --- ## Key Objectives - **Low-resource NER System:** Build a Twi NER system with core entity types: PER, LOC, ORG, DATE,MISC. - **Data Collection & Annotation:** Combine scraping, manual curation, and labeling with clear guidelines. - **Multilingual Baseline:** Experiment with cross-lingual transfer for African language NER. - **Open-source Contribution:** Offer high-quality, reproducible resources for the research community. --- ## Project Structure ``` low-resource-ner-twi/ │ ├── data/ │ ├── raw/ # Raw text data (scraped or collected) │ ├── processed/ # CoNLL files and sentence-level preprocessed data │ └── README.md # Notes on data sources and structure │ ├── scripts/ │ ├── scrape.py # Scraping logic for story/paper content │ ├── split_into_sentences.py # Sentence segmentation for annotation prep │ ├── count_entities.py # Entity/statistics summarizer from .conll │ └── utils.py # Helper functions │ ├── docs/ │ ├── annotation_guide.md # Entity labeling guidelines │ └── research_notes.md # Observations and notes from annotation + modeling │ ├── notebooks/ # Jupyter notebooks for exploration and model training │ ├── models/ # Saved model checkpoints or training configs │ ├── requirements.txt └── READ …