MadaFactBench: A Benchmark for Evaluating Factual Reliability and Hallucination in Large Language Models for Malagasy

Domaine:

natural language processing

Type de record:

dataset

Créateur:

Rat

Éditeur:

Hôte:

MadaFactBench is a benchmark designed to evaluate the factual reliability and hallucination behavior of Large Language Models (LLMs) when answering factual questions in Malagasy. The benchmark contains 500 factual questions written in Malagasy, comprising 175 general-knowledge questions (35%) and 325 questions focusing on Madagascar-specific knowledge (65%). Questions cover multiple domains and difficulty levels and are accompanied by reference answers, accepted answer variants where applicable, and supporting sources. Five publicly accessible conversational AI systems were evaluated on the complete benchmark: GPT-5.6 Luna (ChatGPT), Gemini, Claude Sonnet 5, Meta AI powered by Muse Spark 1.1, and Mistral Large 2 (Le Chat). The experiment produced 2,500 model responses, which were manually evaluated using four behavioral categories: Correct (C), Partially Correct (PC), Hallucination/Factual Error (H), and Abstention (A). The dataset also uses NA for rare technical missing responses caused by collection or alignment issues; these cases are not considered model behavior. The repository provides both the reusable benchmark dataset and the corresponding model responses with their human evaluation labels. MadaFactBench is intended to support research on factuality, hallucination, multilingual LLM evaluation, low-resource languages, and Malagasy Natural Language Processing.

Licenses

Similaires

Probing-Informed Reliability Gates: Multilingual Hallucination Detection and Mitigation in Large Language ModelsUrBLiMP: A Benchmark for Evaluating the Linguistic Competence of Large Language Models in UrduAraLingBench A Human-Annotated Benchmark for Evaluating Arabic Linguistic Capabilities of Large Language ModelsBeyond Fluency: A Rubric-Based Benchmark for Evaluating Saudi Dialect and Cultural Competence in Large Language ModelsEvaluating Large Language Models for Hausa and Fongbe Machine Translation: Benchmarks, Failures, and Metric ReliabilityM3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Models

Probing-Informed Reliability Gates: Multilingual Hallucination Detection and Mitigation in Large Language Models

Large language models can produce fluent but unsupported answers. This paper studies whether probing

UrBLiMP: A Benchmark for Evaluating the Linguistic Competence of Large Language Models in Urdu

Multilingual Large Language Models (LLMs) have shown remarkable performance across various languages

AraLingBench A Human-Annotated Benchmark for Evaluating Arabic Linguistic Capabilities of Large Language Models

We present AraLingBench: a fully human annotated benchmark for evaluating the Arabic linguistic competence of large language models (LLMs). The benchmark spans five core categories: grammar, morphology, spelling, reading comprehension, and syntax, through 150 ex

Beyond Fluency: A Rubric-Based Benchmark for Evaluating Saudi Dialect and Cultural Competence in Large Language Models

Large language models are increasingly deployed in Arabic-speaking markets, yet standard benchmarks

Evaluating Large Language Models for Hausa and Fongbe Machine Translation: Benchmarks, Failures, and Metric Reliability

We investigate the translation quality of current large language models (LLMs) for English-to-Hausa

M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Models

Despite the existence of various benchmarks for evaluating natural language processing models, we ar