goobolabs/SomNLP-Translate
Domain:
natural language processingRecord type:
dataset
Creator:
goo
Host:
SomNLP-Translate is an open, high-quality English–Somali parallel dataset designed to support research and development in machine translation, Natural Language Processing (NLP), Large Language Models (LLMs), and other AI applications for the Somali language.
SomNLP-Translate
Reproducible, quality-controlled English–Somali parallel data for machine translation.
Xog isbarbar-socota oo Ingiriisi iyo Soomaali ah, looguna talagalay turjumaadda mashiinka.
Rust-first data pipeline · streaming · config-driven · full provenance
## Status
SomNLP-Translate has completed **M1 — Foundation**. The data contracts and architecture
are defined; public-source downloaders and processing stages are planned next.
| Milestone | Scope | Status |
|-----------|-------|--------|
| M1 | Workspace, schema, documentation | Done |
| M2 | OPUS and MT560 parallel downloaders | Planned |
| M3 | Cleaning, dual LID, filtering, deduplication | Planned |
| M4 | Leakage-resistant splits and evaluation | Planned |
| M5 | Extended and synthetic sources | Planned |
| M6 | Hugging Face release | Planned |
See PLAN.md and ROADMAP.md.
## Scope
The canonical dataset stores each aligned English–Somali pair once. Language tags make
the record direction-neutral; exporters can create either English→Somali or
Somali→English training views without duplicating source data.
SomNLP-Translate complements
SomNLP-Corpus:
```text
SomNLP-Corpus monolingual Somali documents → language modeling and Somali NLP
SomNLP-Translate aligned English–Somali pairs → translation training and evaluation
```
The repositories follow compatible conventions but remain independent Rust workspaces.
## Planned pipeline
```text
download → merge + exact dedup → clean both sides → dual LID → quality filter
→ near dedup + leakage control → train/dev/test split → final
```
Every stage will stream JSONL, emit statistics, and preserve rejected records with
machine-readable reasons.
## Record format
```json
{
"id": "mt560:sha256-prefix",
"source": {"text": "Peace is important.", "lang": "en"},
"target": {"text": "Nabaddu waa muhiim.", "lang": "so"},
"provenance": {
"source": "mt560",
"collected_at": "2026-07-22T00:00:00Z"
},
"license": "CC-BY-4.0",
"hashes": {
"source": "sha256: …