Chris Emezue, Esther Adenuga

TL;DR

For more than a decade, African-language datasets have been described through scarcity: low-resource languages, limited training data, not enough samples. That description still captures much of the field, but it no longer captures its ceiling. In this analysis, we investigate the largest publicly available African-language text and speech datasets. The current ceiling sits at 9.56 billion words and 18,037 hours of speech. But the numbers are only the beginning. Across more than 2,000 African languages, “large” has at least two dimensions: the volume of data and the number of languages covered. Looking at both reveals a spectrum, describing the unique shape of large African-language datasets, from language-focused collections to datasets approaching continental coverage.

Scarcity is not the whole story

Picture the words “large dataset”. Now picture “African languages”.

For most people, these ideas do not naturally come together. When conversations about AI and datasets turn to Africa, the rhetoric is usually about scarcity: small datasets, limited training data, low-resource languages. Not large.

For a long time, that association was justified. There were not enough sufficiently large datasets to train capable systems for many African languages, much less to pre-train them. Research repeatedly documented these gaps, and the term low-resource became central to how the field understood African-language AI.

But much of that understanding was also built through comparison. African-language datasets were placed beside English and other dominant Western languages and, unsurprisingly, appeared extremely small. English has a dominating effect in almost any comparison of digital language resources. When it becomes the standard against which everything else is measured, scarcity is almost the only conclusion available.

There are two problems with carrying that comparison forward without revisiting it.

The first is time. Comparisons made years ago continue to shape how we think today, even though dataset creation has accelerated across Africa. Community-led initiatives, research projects, public-interest funding and large technology companies have all produced new resources. AI moves quickly, but our mental picture of African-language data has not always moved with it.

The second problem is more fundamental: comparison tells us how African-language datasets look beside English, but it tells us much less about African-language datasets on their own terms. There has been far less work dissecting the largest of these datasets through the multilingual reality of the continent itself. How large are the largest African-language datasets today? What does “large” mean across a landscape of more than 2,000 languages? What happens when a dataset tries to cover one language, ten languages, one hundred languages or one thousand?

This is what we set out to understand.

How we analysed the ceiling

Africa is extraordinarily linguistically and geographically diverse, with more than 2,000 languages. That diversity means that dataset size cannot be understood through volume alone. A dataset containing a billion words in one language and a dataset containing a billion words across hundreds of languages may have the same total volume, but they are not the same kind of resource.

For this analysis, we focused on datasets pertaining to one or more African languages and limited the scope to two modalities: text and speech. We used words as the common unit for text and duration in hours as the common unit for speech. Where a speech dataset included transcripts, we also counted its textual component in the text analysis.

Within each modality and coverage band, we retained the largest 3-5 eligible datasets by volume. As larger datasets were identified, they replaced smaller entries in that band. This allowed us to concentrate on the datasets sitting at the top rather than trying to characterize the entire field.

That distinction is important. This is an analysis of the ceiling, not the field. Many African-language datasets do not appear here because they fall below the top range in their coverage band. Their absence does not mean that they are unimportant or unusable. It simply means that this particular analysis is asking a narrower question: What do the largest publicly available African-language datasets look like?

Our discovery sources included Hugging Face, GitHub and many more sources, made possible through the African AI Atlas. We considered only datasets that were publicly available and accessible; gated datasets were excluded. When one dataset was derived from another, we treated it as a distinct dataset only where the creators had made substantial transformations, such as meaningful translation, preprocessing or restructuring. Simple clones and minimally changed repackagings were excluded.

The underlying data we consulted can be found in one click: https://lanfrica.com/en/atlas?type=large_datasets&largeDatasetSources=lanfrica_insights

How large is large?

The first result is also the simplest: large African-language datasets exist.

We are not saying that every African language now has a large dataset. We are not saying that scarcity has disappeared. We are saying something narrower, but still important: at the ceiling, African-language datasets have reached a scale that should change how we talk about the field.

The largest speech dataset in our analysis contains 18,037 hours of audio. The largest text dataset contains 9.56 billion words and comes from a dataset called finetranslations. These are single datasets and that matters. A single African-language resource contains billions of words; another contains tens of thousands of hours of speech.

Volume alone does not guarantee usefulness. Licensing, quality, duplication, language balance, release status and available compute all affect what a dataset can actually enable. Nor will a very large general-purpose corpus automatically provide the task-specific data needed for every application. But datasets of this size can provide foundational material for pre-training and other large-scale model development. Some of the datasets in our analysis have already been used in the development of major foundation models [1,2,3]. This may partly explain why current foundation models, despite remaining inconsistent across African languages, can generate reasonable text in at least some of them. The open question is whether African builders are organizing around this data and using it deliberately rather than leaving it to others

The current ceiling sits at billions of words and tens of thousands of hours. It may continue moving as dataset creation across the continent accelerates. But even without projecting forward, the present result is clear: large African-language datasets are not a fantasy. They are a fact.

But what does large actually mean for African language datasets?

Africa is not one language context. It is a complex linguistic mosaic. A large “African-language dataset” does not necessarily mean that large amounts of data exist for every African language, or even for many of them. Treating African languages as one entity would reproduce the same simplification we are trying to move beyond.

In most conversations about AI data, large refers almost entirely to volume. More words. More hours. More samples. But African-language datasets introduce an additional dimension that cannot be ignored: language coverage.

This means that largeness has to be examined through

• Breadth: the number of African languages represented in the dataset.
• Depth: the dataset’s total volume, measured in words or hours.

To study breadth, we grouped datasets into five language-coverage bands: one language; 2–10 languages; 10–100 languages; 100–1,000 languages; and 1,000 or more African languages.

Within each modality and coverage band, we retained the largest 3-5 eligible datasets by volume. As larger datasets were identified, they replaced smaller entries in that band. This allowed us to concentrate on the datasets sitting at the top rather than trying to characterize the entire field.

When we plotted the largest datasets across both dimensions, we found an interesting shape.

Text Speech

The spectrum of large African-language datasets. Each point is one of the largest datasets we found, positioned by the number of African languages it covers (breadth) against its volume — words for text, hours for speech. The five shaded bands run from language-focused (a single language) to continental coverage (1,000+ languages); the dashed line connects the largest dataset in each band, allowing us to map a trend across the spectrum. Axes are logarithmic.

To better understand this shape, we characterize it across give categories:

1. language-focused, 
2. targeted multilingual, 
3. multilingual at scale, 
4. the frontier and 
5. continental coverage. 

These categories do not simply describe how many languages a dataset contains. They help explain how the character of largeness changes as language coverage expands.

1. Language focused

Language-focused datasets contain one African language. Here, largeness comes from concentration: all of the dataset’s volume is accumulated around a single language.

For speech, this includes substantial single-language corpora developed for tasks such as automatic speech recognition or text-to-speech. Examples include CLEAR-Global/Luo-Synthetic-ASR-Dataset and BIG-C. For text, the category ranges from task-specific post-training data to large monolingual corpora suitable for language modelling, like UBC-NLP/nilechat-fw-edu-egy. What holds the category together is not one particular task, but the depth built around one language.

This remains an important path for African-language dataset creation. A multilingual dataset is not automatically more useful than a language-focused one. For applications that require dialectal detail, domain depth or community-specific quality, concentration may be precisely the point.

2. Targeted multilingual

Targeted multilingual datasets contain roughly two to ten African languages. They are multilingual, but deliberately bounded.

The languages may be selected because they belong to the same country or region, because a project is working with particular communities, or because the source from which the data is collected supports a defined set of languages. A dataset created in Nigeria may concentrate on Nigerian languages; one developed in Southern Africa may focus on languages from that region. In other cases, the selection is shaped by the languages available through a particular publisher, broadcaster or religious source.

The key property is that multilinguality is targeted rather than open-ended. The creators are not trying to include every possible language. They are building substantial data for a chosen group.

In our speech analysis, the observed ceiling rises from the language-focused band into targeted multilingual datasets. In text, that increase is less consistent. This is an early indication that text and speech do not always scale in the same way.

3. Multilingual at scale

The next category contains approximately 10 to 100 African languages. We call it multilingual at scale because this is where breadth and volume rise together most strongly.

These datasets do not merely include many languages. They include many languages while also accumulating very large amounts of data. In both text and speech, the highest observed ceiling in our analysis sits in this band.

For text, these are often broad, web-scale collections assembled for multilingual training or pre-training, like HuggingFaceFW/finetranslations or castorini/wura. Several contain between one billion and nearly ten billion words. Their size is partly enabled by the fact that more languages contribute more volume, but it also reflects collection methods designed explicitly for scale.

For speech, scale has been built differently. African-language speech cannot simply be crawled from the public web at the same volume as text. Large speech resources therefore tend to depend on deliberate collection: volunteer contribution, compensated recording, institutional partnerships and substantial funding. Efforts such as Common Voice, African Next Voices and WAXAL illustrate different ways of organizing multilingual speech collection at scale.

This category is the apex of the current landscape: the point at which the largest known datasets combine their highest total volume with meaningful multilingual breadth. And yet even 100 languages represent only around five percent of a continent with more than 2,000 languages. The apex is large, but it is not comprehensive.

4. The frontier

It would be reasonable to expect dataset volume to keep rising as more languages are added. That is not what we observe.

Between roughly 100 and 1,000 African languages, the relationship turns. We call this the frontier: the region where expanding language coverage begins to cost volume. The datasets reach more languages, but the observed ceiling no longer increases with them. Instead, it declines. 

On the text side, the frontier still contains large web-crawled or aggregated collections, but their total volume is lower than that of the strongest datasets in the multilingual-at-scale band. Examples include HuggingFaceFW/finepdfs and openbmb/DCAD-2000.

On the speech side, the frontier includes very large aggregation and collection efforts, such as Meta’s Omnilingual ASR Corpus, which brings together speech from hundreds of underserved languages.

The frontier exposes the practical difficulty of multilingual expansion. Every additional language may require new sources, new partnerships, new speakers, new expertise and new quality-control processes. The languages with the smallest digital footprints are also often the hardest to collect at scale. Expanding coverage therefore requires more than repeating the same collection method across a longer list.

Our figure does not prove that adding languages directly causes volume to fall. It shows an observed trade-off among the largest datasets we identified: beyond the multilingual-at-scale band, greater language reach is associated with a lower volume ceiling. The frontier is where that trade-off becomes visible.

5. Continental coverage

The final category contains datasets covering 1,000 or more African languages. We call this continental coverage: the widest linguistic reach in our analysis, but also the lowest observed volume across the five bands.

This category makes the ambiguity of large impossible to ignore. Is a dataset covering more than 1,000 African languages large? 

On the text side, the datasets we found at this level were primarily lexicons, dictionaries and pan-lingual wordlists (e.g. PanLex). They may contain a word or lexical translation equivalent across an extraordinary number of languages, making them valuable as lexical resources or translation seeds. But they are not the same thing as full text corpora containing substantial sentences, documents or discourse in each language.

On the speech side, continental coverage is reached through large-scale aggregation efforts spanning many collections and sources. These resources achieve remarkable breadth, but the amount of speech available for each language can be extremely thin. An example is espnet/mms_ulab_v2.

Continental coverage therefore represents both an achievement and a warning against simple headline numbers. A thousand languages can mean extraordinary reach. It does not necessarily mean substantial training data for a thousand languages.

Large exists. But it has a shape.

Large African-language datasets exist, and we should start thinking of them as large. But we should also understand what that statement means.

The upper end of the landscape is not one undifferentiated mass of data. It is a spectrum. Language-focused datasets build depth around one language. Targeted multilingual datasets concentrate on a selected group. Multilingual-at-scale datasets combine high volume with meaningful breadth. At the frontier, expanding coverage begins to coincide with declining volume. At continental coverage, linguistic reach is at its widest, but data per language is at its thinnest.

This taxonomy gives us a better vocabulary for discussing African-language datasets. Instead of asking only whether a dataset is large, we can ask: large in which dimension? How many languages does it cover? How much data does it contain? How is that volume distributed? What kind of work can it actually support?

Those questions matter for researchers choosing training data, funders deciding what to support, policymakers assessing language inclusion and communities deciding what kind of resources they still need. A billion-word corpus in a few languages and a thousand-language lexicon may both be large, but they solve very different problems.

The low-resource description has not become false. Many African languages remain severely underrepresented, and even the largest datasets can be thin, uneven or unusable for particular purposes. What has changed is that low-resource is no longer a complete description of the landscape. At the upper end, African-language data has reached serious scale. The next task is to understand that scale clearly, use it deliberately and keep watching it.

What this analysis does not tell us

This analysis examines the ceiling, not the whole field. It cannot tell us what the typical African-language dataset looks like or how resources are distributed across all languages. Because we considered only publicly accessible datasets, gated and private resources are absent. Our discovery process may also have missed eligible datasets that were not indexed in the repositories and catalogues we searched.

We also did not evaluate dataset quality. Word counts and speech hours do not capture accuracy, duplication, licensing, demographic balance, dialect coverage, transcription quality or fitness for a particular task. These questions deserve their own analysis, especially because errors and imbalances become harder to see as datasets become larger and more multilingual.

What this analysis offers is a grounded view of the current ceiling: how high it reaches, how it changes across language coverage and why the word large needs more careful interpretation in African-language AI.

About Lanfrica Insights

The Lanfrica Insights team aims to improve public awareness and understanding of AI in Africa. We do this through analysis and evidence-based insights that examine the forces shaping the AI ecosystem on the continent. Through this work, we investigate emerging trends, identify gaps and bottlenecks, and develop scientifically grounded analyses that help make the state of AI in Africa more visible, understandable, and actionable. Learn more about us: https://lanfrica.com/en/insights 

We are accessible via email: insights@lanfrica.com.

Looking for large datasets for a specific African language or task? Every dataset in this analysis — and many more — is tagged and explorable on the Atlas. If you’re building something and want a hand finding the right resource, reach out.

How to cite this work

All visualizations presented in this report are licensed under the CC-BY license, ensuring they are freely available for any use. If you use this analysis or any of the visualizations from this report, please cite it as:

Emezue, C., & Adenuga, E. (2026). The Shape of Large African-Language Datasets. Lanfrica Insights.

@article{lanfrica2026shape,
  title       = {The Shape of Large African-Language Datasets},
  author      = {Emezue, Chris and Adenuga, Esther},
  year        = {2026},
  journal     = {Lanfrica Insights},
  institution = {Lanfrica Labs},
  url         = {https://lanfrica.com/en/blog/insight/the-shape-of-large-african-language-datasets},
  license     = {CC-BY}
}