AfriSpeech/africa-corpus-builder

Get access to monolingual and parallel data for 693 African languages # Africa Corpus Builder A toolkit and small Python library for **retrieving monolingual and parallel text corpora for 693 African languages**. Pull a monolingual corpus for any single African language, or pair one with English, with another African language, or with one of several other world languages. Output is clean, sentence-aligned CSV, ready for machine-translation training and NLP research. The corpora are hosted on HuggingFace at `AfriSpeech/africa-corpus`. The library downloads only the files you actually use and caches them locally, so the repository itself stays lightweight. New languages pushed to the dataset are picked up automatically — no code change or update needed. --- ## What you can build | Corpus type | Example | Notes | |---|---|---| | African ↔ English | Swahili ↔ English | English ships cached; the default target | | African ↔ African | Twi ↔ Yoruba, Hausa ↔ Amharic | align two local languages directly | | African ↔ other language | Twi ↔ French, Zulu ↔ Arabic | French, Arabic, Chinese, Portuguese are cached | | Monolingual | all Swahili sentences | any single language | Every language's text is aligned on a shared verse key, so **any** two languages can be turned into a parallel corpus by a simple join. That is what makes African ↔ African and African ↔ other-language pairs possible. ## Dataset statistics **693 African languages**, **15,974,671 sentences** in total. Sentences = harvested verses available for alignment. | Language | Code | Sentences | |---|---|--:| | Swahili | swh | 356,708 | | Afrikaans | afr | 279,114 | | Shona | sna | 224,417 | | Malgache | plt | 216,384 | | Ganda | lug | 186,020 | | Twi | twi | 154,554 | | Oromo, Arsi / Oromo, Borana-Arsi-Guji | gax | 153,267 | | Amharic | amh | 149,730 | | Masai | mas | 131,770 | | Rwandan | kin | 131,531 | | Nyanja | nya | 130,148 | | Xhosa | xho | 123,914 | | Kikuyu | kik | 122,087 | | Dawro | dwr | 108,043 | | Zulu | zul | 101,217 | | Setswana | tsn | 100,904 | | West Central …