Pegi1727/Diagnosing-Multilingual-Tokenization-Failures-Igbo-Amharic-Russian-English

Diagnosing Multilingual Tokenization Failures: A Typology Aware Framework for Representational Inequality in Large Language Models # Diagnosing-Multilingual-Tokenization-Failures-Igbo-Amharic-Russian-English Diagnosing Multilingual Tokenization Failures: A Typology Aware Framework for Representational Inequality in Large Language Models