freshiwe/On-the-Surface-of-Tone-What-Multilingual-LLMs-Do-and-Don-t-Know-About-Yoruba-Diacritics

Research paper titled: On the Surface of Tone: What Multilingual LLMs Do and Don't Know About Yoruba Diacritics # On the Surface of Tone: What Multilingual LLMs Do and Don't Know About Yorùbá Diacritics A probing study testing whether multilingual language models genuinely represent Yorùbá tone, or merely detect the presence of tone-marking diacritics without understanding what they mean. ## What is this project? (plain-language summary) Yorùbá is a tonal language — the same sequence of letters can mean different things depending on pitch, which is marked in writing using accent marks (tone marks) over vowels. This project asks: when an AI language model reads Yorùbá text, does it actually understand what those tone marks mean, or does it just notice that *some* marks are present without understanding *which* ones? We tested this by taking real Yorùbá sentences and creating damaged versions of each — some with all tone marks removed, some with just one mark deleted, and some with just one mark swapped for the wrong one (a change that leaves the sentence looking almost identical but makes it linguistically incorrect). We then checked whether two AI models could tell correct sentences apart from each type of damage. **Finding:** the models easily detect large-scale tone-mark removal, but perform at chance level (no better than random guessing) when just one mark is swapped for the wrong one. This suggests these models detect the *presence* of tone marking as a surface/orthographic cue, rather than genuinely representing *what the tone marks mean*. ## Repository structure ``` yoruba-tone-probing/ ├── README.md ├── requirements.txt ├── .gitignore ├── LICENSE │ ├── src/ # all analysis code, run in this order │ ├── build_probing_pool.py # Step 1: collect + filter source sentences │ ├── build_tone_pairs.py # Step 2: generate correct/incorrect tone pairs │ ├── week3_probing.py # Step 3: extract hidden states, train layer-wise probes │ ├── week3b_breakdown.py # Step 4: break results down by corruption …