Unkadlabs/somali-token-tax

Somali pays 1.7-2.2x more tokens than English for identical content, across every major LLM tokenizer. Measured on FLORES-200. The tax is a design choice. # The Somali Token Tax **Somali speakers pay 1.7–2.2× more tokens than English speakers for identical content, across every major LLM tokenizer.** More tokens means higher API cost, slower responses, and less room in the context window — a structural disadvantage applied before the model even starts thinking. And it is not a property of the language: a small Somali-first tokenizer makes Somali *cheaper* than English. The tax is a design choice. ## Results FLORES-200 devtest: 1,012 professionally translated parallel sentences, identical meaning in both languages. Token counts over the full set, no special tokens. | Tokenizer | EN tokens | SO tokens | **SO/EN ratio** | |---|---:|---:|---:| | DeepSeek V3 | 27,041 | 60,273 | **2.23×** | | GPT-4 (cl100k) | 27,182 | 60,010 | **2.21×** | | Mistral 7B | 29,997 | 65,813 | **2.19×** | | Llama 3.1 | 27,171 | 59,415 | **2.19×** | | Qwen 2.5 | 27,621 | 60,367 | **2.19×** | | Gemma 2 | 27,036 | 52,557 | **1.94×** | | GPT-4o (o200k) | 26,873 | 46,309 | **1.72×** | | mT5 (multilingual) | 33,791 | 50,359 | **1.49×** | | SomaliWeb (Somali-first, 16k vocab) | 52,618 | 35,867 | **0.68×** | Reading the table: - **Every general-purpose LLM tokenizer taxes Somali at 1.7–2.2×.** The best of them (GPT-4o's o200k, with its larger vocabulary) still charges 72% extra. - **Deliberate multilinguality helps**: mT5's tokenizer, built with balanced multilingual sampling, gets the tax down to 1.49×. - **A Somali-first tokenizer flips the sign.** The SomaliWeb 16k-vocab tokenizer represents the same Somali content in *fewer* tokens than the big tokenizers need for the English (0.68×). Its English column is high, as expected — the point is symmetry: every tokenizer is efficient for what it was built for. Somali was simply not what the major tokenizers were built for. ## Why it matters Tokens are the unit of everything in LLMs: pricing, latency, and context length. At ~2.2× fertility, a Somali user of a per-token API pays roughly double for …