abdullahikawu/indigenous-nlp-benchmark

An open-source NLP benchmark for under-resourced West African languages (Nupe & Gbagyi) featuring diacritic-aware tokenization, Zipfian dynamics, and N-gram language models. # Indigenous NLP Benchmark A university-level low-resource Natural Language Processing assignment focusing on **Nupe** and **Gbagyi** languages. ## Overview This repository serves as a starter template for students learning fundamental NLP concepts through hands-on work with African low-resource languages. The assignment emphasizes: - **Data Collection**: Web scraping and JSON Lines formatting - **Tokenization**: Custom regex-based text processing with diacritic preservation - **Linguistic Analysis**: Zipf's Law and frequency distributions - **Language Modeling**: Bigram models with Laplace smoothing and perplexity evaluation ## Repository Structure ``` indigenous-nlp-benchmark/ ├── README.md # This file ├── requirements.txt # Python dependencies ├── .github/ │ └── workflows/ │ └── autograder.yml # GitHub Actions CI/CD workflow ├── data/ │ ├── nupe/ │ │ ├── raw/ # Raw scraped text data │ │ └── processed/ # Cleaned and tokenized text │ └── gbagyi/ │ ├── raw/ # Raw scraped text data │ └── processed/ # Cleaned and tokenized text ├── submissions/ │ └── group_01_nupe/ │ ├── HW1_assignment.ipynb # Main assignment notebook │ └── group_report.md # Final analysis report └── tests/ ├── autograder_eval.py # PyTest test suite ├── test_nupe_unseen.txt # Unseen test data for evaluation └── test_gbagyi_unseen.txt # Unseen test data for evaluation ``` ## Assignment Overview ### Part 1: Data Collection Implement a web scraper that collects text data from specified URLs and exports to JSON Lines format. **Required Output Format** (`.jsonl`): ```json {"id": 1, "url": "example.com", "date_retrieved": "2024-01-15", "raw_text": "..."} {"id": 2, "url": "example.com", "date_retrieved": "2024-01-15", "raw_text": "..."} ``` ### Part 2: Tokenization & Text …