Logo Lanfrica
  • Accueil
  • Atlas
  • Analyses
  • Documentation
  • Sign in

© 2026 Lanfrica. Tous droits réservés. Tous les droits d'auteur des ressources affichées sur le site Web Lanfrica appartiennent aux détenteurs de droits d'auteur d'origine, sauf indication contraire explicite.

Luganda Tokenizer Evaluation

Domaine:

natural language processing

Type de record:

dataset
Créateur:
Cra
Hôte:
Cross-lingual evaluation of 9 decoder-only LLMs on English (CoNLL-2003, 5,396 sentences) and Luganda (MasakhaNER, 6,055 sentences), quantifying the "tokenizer tax" that low-resource languages pay when using tokenizers designed for English.

Visit

huggingface.co

Languages

Ganda

Tags

lugandatokenizerevaluationlow-resourceafrican-languagesmorphologybpcfertilitycross-lingual

Licenses

apache-2.0

Similaires

Kibalama/luganda-tokenizerrashid0784/luganda-dialect-tokenizermewaeltsegay/tokenizerMehranLM-Tokenizer: A Natively-Trained Tokenizer for Sindhi Language Modelling

Kibalama/luganda-tokenizer

rashid0784/luganda-dialect-tokenizer

mewaeltsegay/tokenizer

Tigrinya Language Tokenizers # Tigrinya Multi-Type Tokenizers for LLM Training A comprehensive col

MehranLM-Tokenizer: A Natively-Trained Tokenizer for Sindhi Language Modelling

Sindhi, an Indo-Aryan language spoken by tens of millions of people, remains severely underrepresent