khaledyusuf44/somali-corpus
Domain:
natural language processingRecord type:
dataset
Creator:
kha
Host:
SomaliWeb v1 — quality-filtered Somali web corpus pipeline. 819K docs / 303M tokens. Dataset: SomaliWeb v1
# somali-corpus — SomaliWeb v1 pipeline
**📄 Paper:** arXiv:2605.18232 — *SomaliWeb v1: A Quality-Filtered Somali Web Corpus with a Matched Tokenizer and a Public Language-Identification Benchmark*
End-to-end pipeline for building **SomaliWeb v1**, a quality-filtered Somali web corpus of ~303M tokens, published on Hugging Face at `khaledyusuf44/somaliweb-v1`.
## What this repo does
Aggregates three public Somali sources (HPLT v2, CC100, Somali Wikipedia) and passes them through six reproducible stages to produce a cleaned, deduplicated, quality-filtered pretraining corpus plus a matched BPE-16K tokenizer.
## Headline numbers
| | |
|---|---:|
| Input (raw aggregated) | 1,372,052 docs · ~588M tokens |
| **Output (SomaliWeb v1)** | **819,322 docs · ~303M tokens** |
| Pipeline retention | 59.7% (40.3% removed as duplicate / short / non-Somali / low-quality) |
| Finding | Number |
|---|---|
| HPLT v2 byte-exact duplicates caught | **17.3% of HPLT input** |
| HPLT v2 docs with mojibake fixable by ftfy | **56.1% of HPLT input** |
| langdetect vs GlotLID v3 on Somali (F1) | 0.884 vs 0.829 (langdetect wins) |
| SomaliWeb v1 tokenizer vs GPT-4 `cl100k_base` on FLORES-200 Somali | **40.2% lower fertility** |
## Repository layout
This repo is **self-contained** — no sibling-repo dependencies. All MinHash + LSH code is inlined under `src/` (originally drafted in `../minhash-dedup-practice/`).
```
somali-corpus/
├── PLAN.md ← full pipeline design doc
├── README.md ← this file
├── requirements.txt ← pinned package versions
├── .gitignore
├── configs/pipeline.yaml ← all knobs (τ, thresholds, seeds)
├── notes/
│ ├── decisions.md ← why each choice was made
│ ├── progress.md ← per-phase execution log with numbers
│ └── learning_plan.md ← 5-level mastery curriculum for presenting
├── src/ ← shared utilities (no sibling-repo imports)
│ ├── n …