Unkadlabs/qor-af-soomaali

Open Somali corpus where every sentence has a named, consenting author — plus the first sentence segmenter and evaluation harness for Somali # Qor Af-Soomaali — the Unkad Somali Corpus An open corpus of Somali text where **every sentence has a named author who consented to its release**, built by Unkad Labs, a non-profit AI research laboratory in Mogadishu. **Data:** huggingface.co · **Platform:** qor.unkad.com · **Licence:** CC BY-SA 4.0 | | | |---|---| | Sentences | 1,547 | | Words | 26,052 | | Documents | 124 | | Domains | 9 | | Contributors | 59 across the platform | ## Why this exists Somali is normally collected by scraping. Corpora built that way carry text of unknown authorship, unknown licence and unknown origin, and low-resource web text is heavily polluted with machine-translation output, which teaches models to reproduce their own errors. This corpus was written rather than found. Every sentence has a known author who agreed to CC BY-SA 4.0 before writing it and chose how to be credited. Nothing here was scraped and nothing was machine translated. That claim is enforced rather than asserted. One contributor's submissions were found to be forwarded messages copied from a Telegram channel and were removed in full, twice, at a cost of roughly 150 sentences and a public milestone. Another who pasted long passages was investigated, gave permission for his own writing directly, and the judgement was recorded against his account with the reviewer's name and the date. Where text could not be licensed it was rejected, however good the Somali was. ## What is in here ``` data/sentences.jsonl one sentence per row, with document_id and position data/train.jsonl one document per row, as it was written CREDITS.md contributors, credited the way each of them chose tools/ the sentence segmenter and its evaluation harness ``` ## The segmentation problem, and a standard for it Counting sentences in Somali has no standard answer, and that is not a detail: it decides every public number and every dataset row. The usual tools do not …