Logo Lanfrica
  • Accueil
  • Atlas
  • Analyses
  • Documentation
  • Sign in

© 2026 Lanfrica. Tous droits réservés. Tous les droits d'auteur des ressources affichées sur le site Web Lanfrica appartiennent aux détenteurs de droits d'auteur d'origine, sauf indication contraire explicite.

XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models

Domaine:

natural language processing

Type de record:

papermodel
Créateur:
LiaGonMaoHou
Hôte:avatar
Large multilingual language models typically rely on a single vocabulary shared across 100+ languages. As these models have increased in parameter count and depth, vocabulary size has remained largely unchanged. This \textit{vocabulary bottleneck} limits the representational capabilities of multilingual models like XLM-R. In this paper, we introduce a new approach for scaling to very large multilingual vocabularies by de-emphasizing token sharing between languages with little lexical overlap and assigning vocabulary capacity to achieve sufficient coverage for each individual language. Tokenizations using our vocabulary are typically more semantically meaningful and shorter compared to XLM-R. Leveraging this improved vocabulary, we train XLM-V, a multilingual language model with a one million token vocabulary. XLM-V outperforms XLM-R on every task we tested on ranging from natural language inference (XNLI), question answering (MLQA, XQuAD, TyDiQA), to named entity recognition (WikiAnn). XLM-V is particularly effective on low-resource language tasks and outperforms XLM-R by 11.2% and 5.8% absolute on MasakhaNER and Americas NLI, respectively. EMNLP 2023

Visit

arxiv.org

Tasks

language modeling

Tags

Computation and LanguageMachine Learning

Similaires

Effective vocabulary expansion of multilingual language models for extremely low-resource languagesScoring Multi-hop Question Decomposition Using Masked Language ModelsAn Exploration of Vocabulary Size and Transfer Effects in Multilingual Language Models for African LanguagesQuantifying the Bias of Transformer-Based Language Models for African American English in Masked Language ModelingProbing Gender Bias in Masked Language Models for Low-Web Data LanguagesMultilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models

Effective vocabulary expansion of multilingual language models for extremely low-resource languages

Multilingual pre-trained language models(mPLMs) offer significant benefits for many low-resource lan

Scoring Multi-hop Question Decomposition Using Masked Language Models

Question answering (QA) is a sub-field of Natural Language Processing (NLP) that focuses on developi

An Exploration of Vocabulary Size and Transfer Effects in Multilingual Language Models for African Languages

Multilingual pretrained language models have been shown to work well on many languages, even those they were not originally pretrained on. Despite their empirical success in downstream tasks, there is still a gap in understanding of "what makes them tick''. In this

Quantifying the Bias of Transformer-Based Language Models for African American English in Masked Language Modeling

International audience In the last three years we witnessed the proliferation of inno

Probing Gender Bias in Masked Language Models for Low-Web Data Languages

Low-resourced languages are increasingly included in large multilingual models. While including more

Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models

Recently, it has been found that monolingual English language models can be used as knowledge bases. Instead of structural knowledge base queries, masked sentences such as "Paris is the capital of [MASK]" are used as probes. We translate the established benchmarks