What Do Prompts Reveal About Model Capabilities in Low-Resource Languages?

Large language models are extremely sensitive to prompt design, a phenomenon which is amplified in multilingual scenarios especially with low-resource languages due to low coverage in model training data, orthographic variation and tokenization issues. In this work, we evaluate a reflective prompt evaluation technique (GEPA) as an inference time optimization strategy on multiple multilingual benchmarks spanning various African languages. Using a more capable model as a reflection model and operating under strict optimization budgets - we show that reflective prompt optimization enables inference time improvements through textual policy evolution resulting in consistent improvements on different tasks without any weight updates. Our evaluation mostly focuses on closed-source models and we observe that on some of the benchmarks, prompt-optimized smaller models can outperform better and more recent models highlighting the importance of instruction design for measuring the capabilities of a language model on a given task. Qualitative review of model outputs before and after optimization also shows that prompt evolution not only reinforces models ability to perform a particular task, it also improves output formatting which is very important for proper model evaluation. We characterize the resulting prompt-latency tradeoff by quantifying the optimization cost in terms of prompt token growth, and our results show that modest increase in prompt size can result in substantial gains in performance. Finally, we argue that benchmark evaluations should report prompt-optimized results alongside baseline prompts in order to properly reflect model capabilities for low-resource languages.