MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs

Domain:

natural language processing

Record type:

paperdataset

Creator:

JumWeiNivBis

Host:

We introduce MultiBLiMP 1.0, a massively multilingual benchmark of linguistic minimal pairs, covering 101 languages and 2 types of subject-verb agreement, containing more than 128,000 minimal pairs. Our minimal pairs are created using a fully automated pipeline, leveraging the large-scale linguistic resources of Universal Dependencies and UniMorph. MultiBLiMP 1.0 evaluates abilities of LLMs at an unprecedented multilingual scale, and highlights the shortcomings of the current state-of-the-art in modelling low-resource languages. Published in TACL, MIT Press

Visit

arxiv.org

Fleurs-SLU: A Massively Multilingual Benchmark for Spoken Language Understanding

Spoken language understanding (SLU) is indispensable for half of all living languages that lack a fo

XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization

The Cross-lingual Natural Language Inference (XNLI) corpus is a crowd-sourced collection of 5,000 test and 2,500 dev pairs for the MultiNLI corpus. The pairs are annotated with textual entailment and translated into 14 languages: French, Spanish, German, Greek, Bu

MVL-SIB: A Massively Multilingual Vision-Language Benchmark for Cross-Modal Topical Matching

Existing multilingual vision-language (VL) benchmarks often only cover a handful of languages. Conse

Minimal pairs 0136-Minimal pairs A description and documentation of Avatime

Elicitation with Sammy, looking for +/- ATR minimal pairs, Ho_SO, anansi_SO. We checked some words t

Massively Multilingual Word Embeddings

We introduce new methods for estimating and evaluating embeddings of words in more than fifty langua

Common Voice: A Massively-Multilingual Speech Corpus

The Common Voice corpus is a massively-multilingual collection of transcribed speech intended for speech technology research and development. Common Voice is designed for Automatic Speech Recognition purposes but can be useful in other domains (e.g. language identi