NjugunaKelvin/africorpus-core

A data infrastructure initiative for african languages # Africorpus Core **A data engineering framework for the ingestion, normalization, and statistical validation of low-resource African language corpora** Overview · The Technical Problem · Architecture · Components · Getting Started --- ## Overview Africorpus Core is an open-source pipeline for turning raw, unstructured text into datasets that are actually usable for machine learning. It is language-agnostic by design: adding a new language requires no code changes. The initial focus is African language corpora across medical, civic, legal, and conversational domains, with an emphasis on provenance and reproducibility over scale for its own sake. This repository holds the pipeline logic only. Raw data, including anything with legal or civic sensitivity, lives in a separate, private repository and never touches this codebase directly. **Scope: text only.** This pipeline ingests, normalizes, and validates written text. Audio recordings, speech datasets, and video transcripts are outside its scope. If your source is audio or video, the expected path is to transcribe it first using a separate ASR step, then feed the resulting text records through this pipeline. --- ## The Technical Problem Most discussion of "low-resource NLP" treats the problem as simply a lack of volume: not enough text, so models underperform. That framing is incomplete, and it leads to the wrong fix (scrape more, regardless of quality). The actual failure modes are more specific: **Tokenizer fertility.** Subword tokenizers (BPE, SentencePiece) are trained predominantly on English and other Indo-European corpora. When applied to morphologically rich, agglutinative languages like Swahili or Amharic, a single semantic word gets fragmented into far more subword tokens than it would in English. This is called high tokenizer fertility, and it has real consequences: longer sequences for the same meaning, higher inference cost, and degraded representation quality, because the model is learnin …