goobolabs/SomNLP-Translate

SomNLP-Translate is an open, high-quality English–Somali parallel dataset designed to support research and development in machine translation, Natural Language Processing (NLP), Large Language Models (LLMs), and other AI applications for the Somali language. SomNLP-Translate Reproducible, quality-controlled English–Somali parallel data for machine translation. Xog isbarbar-socota oo Ingiriisi iyo Soomaali ah, looguna talagalay turjumaadda mashiinka. Rust-first data pipeline · streaming · config-driven · full provenance ## Status SomNLP-Translate has completed **M1 — Foundation**. The data contracts and architecture are defined; public-source downloaders and processing stages are planned next. | Milestone | Scope | Status | |-----------|-------|--------| | M1 | Workspace, schema, documentation | Done | | M2 | OPUS and MT560 parallel downloaders | Planned | | M3 | Cleaning, dual LID, filtering, deduplication | Planned | | M4 | Leakage-resistant splits and evaluation | Planned | | M5 | Extended and synthetic sources | Planned | | M6 | Hugging Face release | Planned | See PLAN.md and ROADMAP.md. ## Scope The canonical dataset stores each aligned English–Somali pair once. Language tags make the record direction-neutral; exporters can create either English→Somali or Somali→English training views without duplicating source data. SomNLP-Translate complements SomNLP-Corpus: ```text SomNLP-Corpus monolingual Somali documents → language modeling and Somali NLP SomNLP-Translate aligned English–Somali pairs → translation training and evaluation ``` The repositories follow compatible conventions but remain independent Rust workspaces. ## Planned pipeline ```text download → merge + exact dedup → clean both sides → dual LID → quality filter → near dedup + leakage control → train/dev/test split → final ``` Every stage will stream JSONL, emit statistics, and preserve rejected records with machine-readable reasons. ## Record format ```json { "id": "mt560:sha256-prefix", "source": {"text": "Peace is important.", "lang": "en"}, "target": {"text": "Nabaddu waa muhiim.", "lang": "so"}, "provenance": { "source": "mt560", "collected_at": "2026-07-22T00:00:00Z" }, "license": "CC-BY-4.0", "hashes": { "source": "sha256: …