bernard-ng/drc-native-tokenizer

Tokenization for Low-Resource Congolese Languages: Efficiency, Coverage, and Downstream Impact # Tokenization for Low-Resource Congolese Languages: Efficiency, Coverage, and Downstream Impact Tokenization is a foundational step in natural language processing (NLP), yet most existing approaches are developed and optimized for high-resource languages, often overlooking African linguistic contexts. This work explores the use of Byte Pair Encoding (BPE) applied exclusively to Congolese native names as training data to investigate whether meaningful subword units can emerge that generalize across the four major national languages of the Democratic Republic of Congo (Lingala, Swahili, Kikongo, and Tshiluba). By constructing a tokenizer solely from personal names—ubiquitous, linguistically rich, and culturally grounded—we aim to examine whether name-derived subword patterns capture phonological and morphological regularities shared across languages. ## Getting Started ### Installation & Setup Instructions and command line snippets bellow are provided to help you set up the project environment quickly and efficiently. assuming you have Python 3.11 and Git installed and working on a Unix-like system (Linux, macOS, etc.). **Using Makefile (Recommended)** ```bash git clone github.com cd drc-native-tokenizer # Setup environment make setup make activate ``` ## Contributors