ritessshhh/NSP

We test LLM comprehension on NSP tasks. Performance drops in low-resource languages. Chain-of-Thought helps weaker models but can hurt stronger ones, showing its effects are complex. # Testing Cross-Lingual Text Comprehension in LLMs Using Next Sentence Prediction This repository contains the codebase for the research paper, "Testing Cross-Lingual Text Comprehension in LLMs Using Next Sentence Prediction". It includes everything from dataset generation to testing all three Large Language Models (LLMs). ## About The Project Large language models show stunning fluency in English, but is this true understanding or just a reflection of massive training data? To find out, we tested their comprehension in a setting where they couldn't rely on data abundance: low-resource languages. Building on prior work (Agarwal et al., AAAI-25) that used Next Sentence Prediction (NSP) as a test, we created a large-scale benchmark with 10,000 questions each for English (a high-resource language), Swahili (medium-resource), and Hausa (low-resource). We then tested several top models, including GPT-4 Turbo, Gemini 1.5 Flash, and LLaMA 3 70B, to see how their comprehension holds up. The results painted a clear picture of how language resources impact performance. While all models excelled in English, their accuracy dropped in Swahili and fell sharply in Hausa, with LLaMA 3 struggling the most. The story became even more interesting when we introduced Chain-of-Thought (CoT) prompting. For the struggling LLaMA 3, CoT acted as a helpful guide, significantly boosting its accuracy. However, for the more capable GPT-4 and Gemini, the same technique often backfired, leading to a kind of "overthinking" that hurt their performance in the cross-lingual context.This reveals that Chain-of-Thought is not a universal solution; its effectiveness depends heavily on the model's baseline capability and the specific context of the task. Our framework pinpoints LLM weaknesses, highlights when CoT helps or hinders cross‑lingual comprehension, and factors influencing their decisions. The goal is to build models that can understand all human languages, not just the most common ones. ## Fil …