Logo Lanfrica
  • Home
  • Atlas
  • Insights
  • Docs
  • Sign in

© 2026 Lanfrica. All rights reserved. All copyrights of the resources shown on the Lanfrica website belong to the original copyright holders, unless explicitly stated otherwise.

Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages

Domain:

natural language processing

Record type:

papermodeldatasetsoftware
Creator:
ImaLinKarSev
Host:avatar
The NLP community has mainly focused on scaling Large Language Models (LLMs) vertically, i.e., making them better for about 100 languages. We instead scale LLMs horizontally: we create, through continued pretraining, Glot500-m, an LLM that covers 511 predominantly low-resource languages. An important part of this effort is to collect and clean Glot500-c, a corpus that covers these 511 languages and allows us to train Glot500-m. We evaluate Glot500-m on five diverse tasks across these languages. We observe large improvements for both high-resource and low-resource languages compared to an XLM-R baseline. Our analysis shows that no single factor explains the quality of multilingual LLM representations. Rather, a combination of factors determines quality including corpus size, script, "help" from related languages and the total capacity of the model. Our work addresses an important goal of NLP research: we should not limit NLP to a small fraction of the world's languages and instead strive to support as many languages as possible to bring the benefits of NLP technology to all languages and cultures. Code, data and models are available at github.com. ACL 2023

Visit

arxiv.org

Tasks

language modeling

Tags

Computation and Language

Similar

EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language ModelsContinually Adding New Languages to Multilingual Language Modelsmultilingual Corpora for Ethiopian LanguagesScaling End-to-End Models for Large-Scale Multilingual ASRUtilizing Multilingual Encoders to Improve Large Language Models for Low-Resource LanguagesScaling Multilingual Language Models and CLCA Score Improvements via Optimal Transport Distillation in MIRACL

EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Models

In this work, we introduce EMMA-500, a large-scale multilingual language model continue-trained on t

Continually Adding New Languages to Multilingual Language Models

Multilingual language models are trained on a fixed set of languages, and to support new languages,

multilingual Corpora for Ethiopian Languages

Scaling End-to-End Models for Large-Scale Multilingual ASR

Building ASR models across many languages is a challenging multi-task learning problem due to large

Utilizing Multilingual Encoders to Improve Large Language Models for Low-Resource Languages

Large Language Models (LLMs) excel in English, but their performance degrades significantly on low-r

Scaling Multilingual Language Models and CLCA Score Improvements via Optimal Transport Distillation in MIRACL

Benefiting from transformer-based pre-trained language models, neural ranking models have made signi