nuel01/yoruba-nlp

BERT-based punctuation restoration and sentence boundary detection for Yoruba — a low-resource West African language # 🌍 Yoruba NLP — Punctuation Restoration & Sentence Boundary Detection BERT-based NLP models for two foundational tasks in **Yoruba text processing**: punctuation restoration and sentence boundary detection. Built to address the critical gap in NLP tooling for one of Africa's most widely spoken languages. --- ## 🔗 Resources | Resource | Link | |---|---| | 🤗 Punctuation Model | abnuel/yoruba_task1_punctua… | | 🤗 Sentence Boundary Model v1 | abnuel/yoruba_sent_boundary | | 🤗 Sentence Boundary Model v2 | abnuel/yoruba_sent_boundary… | | 📦 Dataset | abnuel/yor_punctuation | --- ## 📌 Why Yoruba NLP? Yoruba is spoken by approximately **40–50 million people**, primarily in southwestern Nigeria and the diaspora. Despite this, it remains severely underrepresented in NLP research and tooling. Two practical problems motivated this work: 1. **Unpunctuated text is ubiquitous** — social media posts, transcribed speech, and scanned documents in Yoruba typically lack consistent punctuation, which breaks downstream NLP pipelines. 2. **Sentence segmentation is unreliable** — standard sentence tokenizers trained on English (or even other African languages) perform poorly on Yoruba due to its tonal structure and morphological complexity. This project builds the foundational preprocessing tools needed to enable higher-level Yoruba NLP tasks. --- ## 🗂️ Project Structure ``` yoruba-nlp/ ├── README.md ├── requirements.txt ├── notebooks/ │ ├── 01_data_exploration.ipynb # Dataset analysis and statistics │ ├── 02_punctuation_finetuning.ipynb # Punctuation restoration training │ └── 03_sentence_boundary_finetuning.ipynb # SBD training ├── src/ │ ├── train.py # Token classification training script │ ├── inference.py # Inference utilities for both tasks │ ├── evaluate.py # F1, precision, recall per label │ └── preprocess.py # Data preprocessing and label …