Aymenec-212/darija-tokenizer

# How GPT-2's tokenizer handles Moroccan Darija In this notebook, we measure BPE tokenizer's compression rate on Moroccan Darija. **Finding:** GPT-2's BPE provides no meaningful compression over Darija. Its merge process reached the character level (one token = one character). the first logical hypothesis is it never accumulated enough frequency of darija words to build subword or word units. On English, the same encoder reaches whole-word merges. ## Results | Measurement | Darija | English | Ratio | |---|---|---|---| | Tokens per word | 5.91 | 1.30 | 4.5× | | Characters per token | 1.02 | 4.30 | 4.2× | | Unique token types (20,479 chars) | 110 | 1,416 | 12.9× | | Token stream that is byte fragments | 46.4% | ~0% | — | ## Why this happens GPT-2's BPE operates on UTF-8 bytes, and its merges were learned by frequency on WebText, which is overwhelmingly English. Arabic characters occupy two bytes each. Arabic sequences appeared often enough for the merge process to reassemble *individual characters* from their byte pairs — and only for part of them, since 46% of the emitted token stream is still incomplete UTF-8. It never got further. The practical cost is threefold: ~4.5× sequence length for equivalent content, a proportionally smaller effective context window, and model capacity spent learning to compose characters into morphemes before any semantics can be learned. ## Null results **Merge-order position — no effect.** Hypothesis: Darija tokens should cluster at low token IDs, since GPT-2 assigns IDs in merge order and rare pairs merge late. Observed medians were 3,686 (Darija) and 3,887 (English) — within ~5%. Plausible explanation: the D8/D9 lead bytes are shared across *all* Arabic script, so those merges were learned early despite Arabic being rare in the training corpus. **Embedding norms — real but small.** Darija-reachable tokens have mean L2 norm 3.355 (sd 0.517, n=430) against 3.504 (sd 0.471, n=1,379) for English-reachable tokens. Statistically …