miloudbelarebia/arabic-tax

Arabic Costs Double on AI — why every Arabic-script language (Darija, MSA, Persian, Urdu) pays a 2x token tax on ChatGPT, Claude & Gemini. # arabic-tax > **The same sentence costs 2× more tokens in Arabic than in English on AI.** > Not an opinion. The mechanical result of two design choices made in 1992 and 2019. > Affects every Arabic-script language — Darija, Algerian, Egyptian, MSA, Persian, Urdu — across **ChatGPT, Claude, Gemini**, and every major LLM. --- ## TL;DR I downloaded OpenAI's official tokenizer file (`o200k_base`, 200,019 entries, used by GPT-4o / GPT-4.1 / GPT-5.x / o1 / o3 — verified on platform.openai.com, all return identical token counts). I measured exactly how it splits the vocabulary by writing system. **Latin gets 67.4% of the dictionary. Arabic gets 4.0%. The ratio is 17 to 1.** Direct consequence: the same idea takes about twice as many tokens in Arabic as in English on ChatGPT — more API cost, smaller usable context window, slower answers, worse output quality. This repo holds the **reproducible Python notebook**, the **infographic**, and the **historical analysis** of why this happens. --- ## Why does this happen? A 32-year-old design story This isn't a bug. It's the **stacked consequence of two design choices** — one from 1992, one from 2019. Both were defensible at the time. Neither was malicious. But together, they produce today's tax on Arabic-script users. ### 🧱 Layer 1 — UTF-8 (1992): the foundation **What it is.** UTF-8 is the standard that decides how text is stored as bytes on disk. Every text file, every webpage, every API call uses it. It was designed by Ken Thompson and Rob Pike in September 1992, on a placemat in a New Jersey diner. **The trade-off they made.** UTF-8 had to stay 100% backward-compatible with ASCII (the old 128-character English-only standard from the 1960s). So the design was: the 128 ASCII characters keep their old 1-byte encoding. Everything else needs more bytes. **The mechanical result, as written in RFC 3629:** | Unicode range | Byte cost | Examples | |---|---:|---| | U+0000 – U+007F | **1 byte** | `A`, `z`, digit …