Analyzing In-Context Language Learning in Long-Context Large Language Models

We evaluated the in-context learning capabilities of long-context large language models for machine translation of Kalamang, an extremely low-resource Papuan language of western New Guinea. Using Qwen2.5-7B-Instruct-1M and Llama-3.1-8B-Instruct for translation generation across Kalamang↔English directions, we evaluated two strategies for leveraging linguistic resources: (1) LLM-based translation quality assessment with Qwen3-8B as judge and the COMET metric, and (2) prompt augmentation with synthetic parallel sentences. Evaluations with Qwen3 as judge showed that neither pointwise nor pairwise ranking strategies improved baseline chrF scores, with the best performance (approximately 32 chrF for English → Kalamang) achieved without assessment. The COMET metric exhibited unreliable correlation with actual accuracy, showing Pearson correlation coefficients of 0.44 for English → Kalamang but −0.77 for Kalamang → English, with standard deviations on synthetic data substantially higher than chrF (ranging from 9.3–23.2 versus 6.3–23.4). Prompt augmentation with synthetic data across two generation attempts produced improvements rarely exceeding 4.19 chrF points. At the same time, degradations frequently reached 6.14 points, with per-sentence analysis showing only 26 of 50 test sentences improved and high variance (standard deviation of 9.19 points) among improved examples. Direct comparison of real versus synthetic sentences revealed consistent performance gaps of 8–13 chrF points across all context settings, with synthetic-only performance ranging from 16.5–24.6 for Kalamang → English compared to 29.2–32.7 with real sentences. Extended context (50–375 reference sentences) paradoxically degraded performance by up to 11 chrF points compared to optimal configurations (10–50 references), contradicting claims of benefit from scaling context length. These findings indicate that current long-context LLMs neither effectively leverage extended contextual information nor reliably assess translation quality for extremely low-resource languages. Progress in this domain will likely depend on improved cross-lingual transfer learning and retrieval mechanisms rather than on optimizing in-context learning examples alone.