Logo Lanfrica
  • Home
  • Atlas
  • Insights
  • Docs
  • Sign in

© 2026 Lanfrica. All rights reserved. All copyrights of the resources shown on the Lanfrica website belong to the original copyright holders, unless explicitly stated otherwise.

afri-fertility: African Language Tokenization Measurement Results

Domain:

natural language processing

Record type:

dataset
Creator:
Cip
Host:
Measurement dataset for The African Language Tax — the first systematic audit of the subword tokenization penalty imposed on African languages by frontier large language models. Every row is one (language, tokenizer, corpus) triple, with fertility, English-relative premium, and confidence intervals computed from a parallel corpus using sum-then-divide aggregation. Property Value

Visit

huggingface.co

Tasks

language modeling

Languages

AfrikaansAkanAmharicBamanankanGandaHausaIgboKinyarwandaLingalaN’ko+10

Tags

tokenizationafrican-languagesnlpsubword-fertilityllmmultilingualai-equity

Licenses

apache-2.0

Similar

CipherSenseAI/afri-fertilityTokenization Matters: Navigating Data-Scarce Tokenization for Gender Inclusive Language TechnologiesAfri-XNLI+: Extending the XNLI Dataset with New African Language Translationsmohamad-755/african-tokenization-superbpemohamad-755/magnet-african-tokenizationLong-term fertility experiments for irrigated rice in the West African Sahel: agronomic results

CipherSenseAI/afri-fertility

Measure the tokenization tax on African languages. afri-fertility quantifies how many more tokens an

Tokenization Matters: Navigating Data-Scarce Tokenization for Gender Inclusive Language Technologies

Gender-inclusive NLP research has documented the harmful limitations of gender binary-centric large

Afri-XNLI+: Extending the XNLI Dataset with New African Language Translations

Afri-XNLI+ is a multilingual dataset consisting of translated premise-hypothesis sentence pairs from

mohamad-755/african-tokenization-superbpe

Comparing SuperBPE against standard BPE tokenization across 9 African languages (Amharic, Hausa, Igb

mohamad-755/magnet-african-tokenization

Reproducing and extending MAGNET (Ahia et al., NeurIPS 2024) to test multilingual tokenization fairn

Long-term fertility experiments for irrigated rice in the West African Sahel: agronomic results

Long-term fertility experiments (LTFEs) are a tool to investigate the sustainability of cropping sys