Rethinking Multilingual Continual Pretraining: Data Mixing for Adapting LLMs Across Languages and Resources

Domain:

natural language processing

Record type:

paper

Creator:

Li,Ji,LuoTie

Host:

Large Language Models (LLMs) exhibit significant disparities in performance across languages, primarily benefiting high-resource languages while marginalizing underrepresented ones. Continual Pretraining (CPT) has emerged as a promising approach to address this imbalance, although the relative effectiveness of monolingual, bilingual, and code-augmented data strategies remains unclear. This study systematically evaluates 36 CPT configurations involving three multilingual base models, across 30+ languages categorized as altruistic, selfish, and stagnant, spanning various resource levels. Our findings reveal three major insights: (1) Bilingual CPT improves multilingual classification but often causes language mixing issues during generation. (2) Including programming code data during CPT consistently enhances multilingual classification accuracy, particularly benefiting low-resource languages, but introduces a trade-off by slightly degrading generation quality. (3) Contrary to prior work, we observe substantial deviations from language classifications according to their impact on cross-lingual transfer: Languages classified as altruistic often negatively affect related languages, selfish languages show conditional and configuration-dependent behavior, and stagnant languages demonstrate surprising adaptability under certain CPT conditions. These nuanced interactions emphasize the complexity of multilingual representation learning, underscoring the importance of systematic studies on generalizable language classification to inform future multilingual CPT strategies. COLM 2025

Visit

arxiv.org

Tasks

transfer learning

Leveraging LLMs for Synthesizing Training Data Across Many Languages in Multilingual Dense Retrieval

There has been limited success for dense retrieval models in multilingual retrieval, due to uneven a

Adapting Multilingual LLMs to Low-Resource Languages with Knowledge Graphs via Adapters

This paper explores the integration of graph knowledge from linguistic ontologies into multilingual

Teaching LLMs to Abstain across Languages via Multilingual Feedback

Multilingual LLMs often have knowledge disparities across languages, with larger gaps in under-resou

Framing Political Bias in Multilingual LLMs Across Pakistani Languages

Large Language Models (LLMs) increasingly shape public discourse, yet most evaluations of political

Adapting Multilingual Speech Representation Model for a New, Underresourced Language through Multilingual Fine-tuning and Continued Pretraining

In recent years, neural models learned through self-supervised pretraining on large scale multilingu

Rethinking Full Finetuning from Pretraining Checkpoints in Active Learning for African Languages

Active learning (AL) aims to reduce annotation effort by iteratively selecting the most informative