khaledyusuf44/somali-corpus

SomaliWeb v1 — quality-filtered Somali web corpus pipeline. 819K docs / 303M tokens. Dataset: SomaliWeb v1 # somali-corpus — SomaliWeb v1 pipeline **📄 Paper:** arXiv:2605.18232 — *SomaliWeb v1: A Quality-Filtered Somali Web Corpus with a Matched Tokenizer and a Public Language-Identification Benchmark* End-to-end pipeline for building **SomaliWeb v1**, a quality-filtered Somali web corpus of ~303M tokens, published on Hugging Face at `khaledyusuf44/somaliweb-v1`. ## What this repo does Aggregates three public Somali sources (HPLT v2, CC100, Somali Wikipedia) and passes them through six reproducible stages to produce a cleaned, deduplicated, quality-filtered pretraining corpus plus a matched BPE-16K tokenizer. ## Headline numbers | | | |---|---:| | Input (raw aggregated) | 1,372,052 docs · ~588M tokens | | **Output (SomaliWeb v1)** | **819,322 docs · ~303M tokens** | | Pipeline retention | 59.7% (40.3% removed as duplicate / short / non-Somali / low-quality) | | Finding | Number | |---|---| | HPLT v2 byte-exact duplicates caught | **17.3% of HPLT input** | | HPLT v2 docs with mojibake fixable by ftfy | **56.1% of HPLT input** | | langdetect vs GlotLID v3 on Somali (F1) | 0.884 vs 0.829 (langdetect wins) | | SomaliWeb v1 tokenizer vs GPT-4 `cl100k_base` on FLORES-200 Somali | **40.2% lower fertility** | ## Repository layout This repo is **self-contained** — no sibling-repo dependencies. All MinHash + LSH code is inlined under `src/` (originally drafted in `../minhash-dedup-practice/`). ``` somali-corpus/ ├── PLAN.md ← full pipeline design doc ├── README.md ← this file ├── requirements.txt ← pinned package versions ├── .gitignore ├── configs/pipeline.yaml ← all knobs (τ, thresholds, seeds) ├── notes/ │ ├── decisions.md ← why each choice was made │ ├── progress.md ← per-phase execution log with numbers │ └── learning_plan.md ← 5-level mastery curriculum for presenting ├── src/ ← shared utilities (no sibling-repo imports) │ ├── n …