Lost in Execution: On the Multilingual Robustness of Tool Calling in Large Language Models

Domaine:

natural language processing

Type de record:

paperdataset

Créateur:

LuoKutOkoXu,

Hôte:

Large Language Models (LLMs) are increasingly deployed as agents that invoke external tools through structured function calls. While recent work reports strong tool-calling performance under standard English-centric evaluations, the robustness of tool calling under multilingual user interactions remains underexplored. In this work, we introduce MLCL, a diagnostic benchmark, and conduct a systematic evaluation of multilingual tool calling across Chinese, Hindi, and the low-resource language Igbo. Through fine-grained error analysis, we show that many failures occur despite correct intent understanding and tool selection. We identify parameter value language mismatch as a dominant failure mode, where models generate semantically appropriate parameter values in the user's language, violating language-invariant execution conventions. We further evaluate several inference-time system strategies and find that while these strategies substantially reduce language-induced execution errors, none of them can fully recover English-level performance.

Visit

arxiv.org

Similaires

IndicSafeEval: Safety Robustness of Large Language Models under Multilingual Persuasive Jailbreak AttacksQuantifying Language Disparities in Multilingual Large Language ModelsBridging language gaps in multilingual large language modelsHealMed: Multilingual Evaluation of Large Language Models in MedicineIsolating Culture Neurons in Multilingual Large Language ModelsMultilingual Emotion Neurons in Large Audio-Language Models

IndicSafeEval: Safety Robustness of Large Language Models under Multilingual Persuasive Jailbreak Attacks

Large language models (LLMs) are increasingly used in multilingual settings, yet their safety is sti

Quantifying Language Disparities in Multilingual Large Language Models

Results reported in large-scale multilingual evaluations are often fragmented and confounded by fact

Bridging language gaps in multilingual large language models

Large language models (LLMs) have revolutionized natural language processing, yet significant perfor

HealMed: Multilingual Evaluation of Large Language Models in Medicine

We present HealMed, an expert-reviewed benchmark for multilingual evaluation of large language model

Isolating Culture Neurons in Multilingual Large Language Models

Language and culture are deeply intertwined, yet it has been unclear how and where multilingual larg

Multilingual Emotion Neurons in Large Audio-Language Models

Emotion is central to human communication, and its expression varies across languages. Large audio-l