goobolabs/SomNLP-Corpus

SomNLP-Corpus is an open, high-quality Somali text corpus designed to support research and development in Natural Language Processing (NLP), Large Language Models (LLMs), and other AI applications for the Somali language. SomNLP-Corpus A high-quality, scalable, reproducible Somali text corpus for NLP, LLMs, and AI research. Qoraal Soomaali nadiif ah oo loogu talagalay cilmi-baarista AI iyo NLP-ga. Rust-first data pipeline · streaming · config-driven · full provenance Status · What we built · Corpus results · Pipeline · Tokenizer · Quick start · Sources · Docs --- ## Status | Phase | Scope | Status | |-------|-------|--------| | 1 — Foundation | Workspace, shared types | ✅ Done | | 2 — Public datasets | Six downloaders + merge | ✅ Done | | 3 — Processing pipeline | Clean → LID → deep clean → near-dedup | ✅ Done | | 4 — Collection | Wikipedia, web scraping | 🔜 Next | | 5 — Release | Hugging Face packaging | Planned | **Track A is live:** download six public Somali datasets, merge, clean, verify language, and deduplicate into a training-ready corpus. **Track B next:** Wikipedia and targeted Somali web collection. See ROADMAP.md and PLAN.md. ## What we built - **Six downloaders** — HPLT, CC100, mC4, OPUS, MADLAD, MT560 - **Five processing stages** — merge + exact dedup, clean, LID (`lingua`), deep clean (v0.2), near-dedup (MinHash + LSH) - **`CorpusRecord` metadata** — provenance, content hash, dedup info, quality flags on every kept line - **Reject sidecars** — full text + reason for every dropped record; inspect with `reports/inspect_drops.sh` - **Single config** — `configs/pipeline.toml` ``` SomNLP ── SomNLP-Corpus (this repo) → Translate · NER · QA · Instruct · Sentiment · Bench ``` ## Corpus results Full 6-source run (HPLT, CC100, mC4, OPUS, MADLAD, MT560) through the **v0.2 pipeline** (merge → clean → LID → deep clean → near-dedup). Document counts are from per-stage stats; **final word count is measured** on `data/final/final_so.jsonl`. Intermediate word/token figures use the final corpus average (~317 words/doc) and are marked with ~. v0.1 baseline (without deep clean): 1.77M docs · 591M words — see docs/CLEANING_STRATEGY.md. | Stage | Documents | Word …