Unkadlabs/qor-hubi-floor

Can open language models tell a real Somali sentence from its own words shuffled? Three models, 2,400 judgements, two failed predictions. # qor-hubi-floor Can an open language model do the easiest version of our community's validation job: tell a real Somali sentence from the same words in a random order? Three models, 200 verified Somali sentences from unkadlabs/qor-af-soomaali, 200 English control sentences, all shuffled counterparts, 2,400 judgements. Everything ran locally on 4-bit quantised weights on one machine in Mogadishu. ## Result | | English | Somali | real Somali rejected | scrambled Somali accepted | |---|---|---|---|---| | gemma2:9b | 0.985 | 0.935 | 5% | 8% | | llama3.1:8b | 0.950 | 0.825 | 31% | 4% | | qwen2.5:7b | 0.978 | 0.542 | 1% | 90% | Two of our three pre-registered predictions failed. See PREDICTIONS.md for what we committed to before running and ARTICLE.md for the writeup, including the finding that 26 points of qwen's failure is threshold placement rather than missing information. The single number hides two opposite failures. Llama discards real contributions; Qwen waves scrambled text through. Scoring the models without using their yes-or-no answer shows the information is present and the decision threshold is what discards it. ## Method in one paragraph Each item is one sentence and one question: is this a coherent, natural sentence with correct word order, yes or no. We never parse generated prose. We request a single token and read the model's probability mass over Yes-variants against No-variants, so refusals and verbosity cannot corrupt the measurement. Every model is served identically through Ollama's OpenAI-compatible endpoint, which is what makes the cross-model comparison valid. Sampling is seeded; reruns skip completed items. ## Reproducing Requires Ollama and Python 3.12. ```bash ollama pull qwen2.5:7b && ollama pull gemma2:9b && ollama pull llama3.1:8b pip install datasets matplotlib # fetch the corpus the eval set is built from curl -L huggingface.co \ -o data/sent …