On the Calibration of Massively Multilingual Language Models

Domain:

natural language processing

Record type:

paper

Creator:

AhuSitDanCho

Host:

Massively Multilingual Language Models (MMLMs) have recently gained popularity due to their surprising effectiveness in cross-lingual transfer. While there has been much work in evaluating these models for their performance on a variety of tasks and languages, little attention has been paid on how well calibrated these models are with respect to the confidence in their predictions. We first investigate the calibration of MMLMs in the zero-shot setting and observe a clear case of miscalibration in low-resource languages or those which are typologically diverse from English. Next, we empirically show that calibration methods like temperature scaling and label smoothing do reasonably well towards improving calibration in the zero-shot scenario. We also find that few-shot examples in the language can further help reduce the calibration errors, often substantially. Overall, our work contributes towards building more reliable multilingual models by highlighting the issue of their miscalibration, understanding what language and model specific factors influence it, and pointing out the strategies to improve the same. EMNLP 2022

Visit

arxiv.org

Tasks

language modeling

SERENGETI: Massively Multilingual Language Models for Africa

Multilingual language models (MLMs) acquire valuable, generalizable linguistic information during pretraining and have advanced the state of the art on task-specific finetuning. So far, only ~ 28 out of ~2,000 African languages are covered in existing language mode

EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Models

In this work, we introduce EMMA-500, a large-scale multilingual language model continue-trained on t

Massively Multilingual Adaptation of Large Language Models Using Bilingual Translation Data

This paper investigates a critical design decision in the practice of massively multilingual continu

GlotEval: A Test Suite for Massively Multilingual Evaluation of Large Language Models

Large language models (LLMs) are advancing at an unprecedented pace globally, with regions increasin

Babel-ImageNet: Massively Multilingual Evaluation of Vision-and-Language Representations

Vision-and-language (VL) models with separate encoders for each modality (e.g., CLIP) have become the go-to models for zero-shot image classification and image-text retrieval. The bulk of the evaluation of these models is, however, performed with English text only:

Massively Multilingual Word Embeddings

We introduce new methods for estimating and evaluating embeddings of words in more than fifty langua