Logo Lanfrica
  • Home
  • Atlas
  • Insights
  • Docs
  • Sign in

© 2026 Lanfrica. All rights reserved. All copyrights of the resources shown on the Lanfrica website belong to the original copyright holders, unless explicitly stated otherwise.

Language Models' Factuality Depends on the Language of Inquiry

Domain:

natural language processing

Record type:

paperdataset
Creator:
AggTanAgrAyu
Host:avatar
Multilingual language models (LMs) are expected to recall factual knowledge consistently across languages, yet they often fail to transfer knowledge between languages even when they possess the correct information in one of the languages. For example, we find that an LM may correctly identify Rashed Al Shashai as being from Saudi Arabia when asked in Arabic, but consistently fails to do so when asked in English or Swahili. To systematically investigate this limitation, we introduce a benchmark of 10,000 country-related facts across 13 languages and propose three novel metrics: Factual Recall Score, Knowledge Transferability Score, and Cross-Lingual Factual Knowledge Transferability Score-to quantify factual recall and knowledge transferability in LMs across different languages. Our results reveal fundamental weaknesses in today's state-of-the-art LMs, particularly in cross-lingual generalization where models fail to transfer knowledge effectively across different languages, leading to inconsistent performance sensitive to the language used. Our findings emphasize the need for LMs to recognize language-specific factual reliability and leverage the most trustworthy information across languages. We release our benchmark and evaluation framework to drive future research in multilingual knowledge transfer.

Visit

arxiv.org

Languages

Swahili

Tags

Computation and LanguageArtificial Intelligence

Similar

On the Calibration of Massively Multilingual Language ModelsOn the Universality of Deep Contextual Language ModelsOn the Utility of Pretraining Language Models on Synthetic DataA Qualitative Inquiry into the South African Language Identifier’s Performance on YouTube Comments.On Language Models for CreolesEvaluation of Arabic Large Language Models on Moroccan Dialect

On the Calibration of Massively Multilingual Language Models

Massively Multilingual Language Models (MMLMs) have recently gained popularity due to their surprisi

On the Universality of Deep Contextual Language Models

Deep Contextual Language Models (LMs) like ELMO, BERT, and their successors dominate the landscape o

On the Utility of Pretraining Language Models on Synthetic Data

Development of pre-trained language models has predominantly relied on large amounts of datasets. Ho

A Qualitative Inquiry into the South African Language Identifier’s Performance on YouTube Comments.

On Language Models for Creoles

Creole languages such as Nigerian Pidgin English and Haitian Creole are under-resourced and largely

Evaluation of Arabic Large Language Models on Moroccan Dialect

Large Language Models (LLMs) have shown outstanding performance in many Natural Language Processing