kuenane/sesotho-llm-prompting

Prompt engineering findings for Sesotho (Southern Sotho) using Claude — from a production remittance system # sesotho-llm-prompting Prompting LLMs in Sesotho: Findings from a Production System Author: Lebohang Kuenane — @kuenane Context: PayPesa — Lesotho's first PayPal-to-M-Pesa remittance app Model tested: claude-3-opus-20240229 Task: Generating SMS notifications in Sesotho for remittance recipients Status: Ongoing — findings updated as the system evolves Why This Exists Sesotho (Sesotho sa boroa/ Southern Sotho) is a Bantu language spoken by approximately 5–6 million people, primarily in Lesotho and South Africa. It is severely underrepresented in the training corpora of large language models. When I integrated Claude into PayPesa to generate recipient SMS notifications, I assumed language control would be straightforward — specify the language, get the output. It was not. What followed was months of iterative prompt engineering against a model that had clearly seen far less Sesotho than English, and the failures were not random. They were patterned, reproducible, and instructive. This document records what I found: the failure modes, the prompt architectures I tested, what worked, what didn't, and the open questions I haven't resolved yet. This is not a benchmark study. I had no labelled ground-truth dataset, no native-speaker annotation pipeline, and no compute budget. What I had was a production system, real users, and my own native fluency in Sesotho as the evaluation instrument. The Task Each time a PayPesa transfer completes (or is queued), the system generates an SMS notification for the recipient. Requirements: Written in Sesotho (not English, not a mix) 160 characters or fewer (standard SMS limit) Correct recipient name, amount in LSL (Lesotho Loti), and sender reference Appropriate register: warm but professional, not robotic No hallucinated amounts — the loti figure must match the input exactly A typical target output looks like: Lumela [Name]! O amohetse LSL [amount] ho tsoa ho [sender]. Lefa e fihlile. - PayPesa Failure Modes Observed 1. Wrong Grammar and …