CipherSenseAI/afri-fertility

Measure the tokenization tax on African languages. afri-fertility quantifies how many more tokens and dollars frontier LLMs charge for African languages compared to English. This is the measurement engine behind the African Language Tax paper. # afri-fertility **Measure the tokenization "tax" on African languages.** Commercial LLMs bill, throttle, and context-budget per token. Because the same meaning takes more tokens in African languages than in English, speakers and builders face a structural cost, latency, and context penalty — before the model is even invoked. `afri-fertility` measures that penalty precisely. It is the open measurement engine behind *The African Language Tax* paper, the public **Token Fertility Leaderboard**, and the cost-calculator widget at datalens.africa. ``` afri-fertility reproduce ``` ``` Tokenizer Language Fertility Premium ────────────────────────────────────────────────── openai/o200k_base amh 8.500 7.83× openai/o200k_base yor 2.674 2.46× openai/o200k_base swh 1.800 1.66× openai/o200k_base fra 1.265 1.17× ``` --- ## Install ```bash pip install afri-fertility # core: tiktoken + HF backends pip install "afri-fertility[api]" # + Claude / Gemini count-only pip install "afri-fertility[viz]" # + matplotlib figures pip install "afri-fertility[dev]" # + pytest, hypothesis ``` Requires Python 3.11+. The core path is CPU-only and key-free. --- ## Quickstart ### Single-text measurement ```python from afri_fertility import measure_text m = measure_text("Àwọn ará Nàìjíríà", tokenizer="openai/o200k_base") print(f"tokens={m.tokens} fertility={m.fertility:.2f} cpt={m.cpt:.2f}") # tokens=8 fertility=2.67 cpt=2.25 ``` ```bash afri-fertility measure \ --text "Àwọn ará Nàìjíríà tó ń gbé ní ìlú Èkó" \ --lang yor \ --models openai/o200k_base,openai/cl100k_base ``` ### Cost calculator ```python from afri_fertility import cost_of results = cost_of("Àwọn ará Nàìjíríà", lang="yor", models=["openai/o200k_base"]) for r in results: print(f"{r.tokenizer}: ${r.total_cost_usd:.6f} NGN {r.costs_local['NGN']:.4f}") ``` ```bash afri-fertility cost \ --text "Àwọn ará Nàìjíríà" \ --lang yor \ --models o …