arcange9/munyarwanda-ai-data-pipeline

Munyarwanda AI v0.3 - Kinyarwanda dataset discovery, cleaning, dedup, mixing & QLoRA training pipeline (Colab-ready) # Munyarwanda AI — Data Pipeline Production data acquisition, cleaning, validation, deduplication, mixing and training pipeline for **Munyarwanda AI**, a Kinyarwanda-first LLM. - Dataset: huggingface.co (creates config `v0.3`) - Model target: huggingface.co - Colab notebook: `notebooks/Munyarwanda-AI-v0.3-Full-Data-Pipeline.ipynb` ## What's here ``` data/ resources_manifest.json 43 verified resources + licenses (research-backed) raw_manifests/ download manifests (raw data never committed) processed/ cleaned → dedup → final datasets (runtime output) reports/ discovery / validation / language / dedup / mixture / bench reports configs/ data_mix_v0.3.yaml mixture targets & ratios (rationale documented inline) training_v0.3.yaml explicit training hyperparameters (QLoRA on Qwen3-0.6B) scripts/ discover_resources.py re-verify manifest against live APIs download_datasets.py fetch/stream approved datasets validate_datasets.py schema + license gate common.py Kinyarwanda language detector + quality scoring clean_text.py normalization, language ID, provenance language_filter.py language distribution audit deduplicate.py exact hash + MinHash LSH (cross-dataset) build_pretraining_corpus.py / build_instruction_data.py / build_translation_data.py / build_eval_data.py / build_dataset_mix.py notebooks/ Munyarwanda-AI-v0.3-Full-Data-Pipeline.ipynb 29-step end-to-end run docs/ KINYARWANDA_DATASET_CATALOG.md full 43-resource catalog w/ licenses & tiers DATA_LICENSES.md license policy + attribution block DATA_PIPELINE.md architecture walkthrough DATASET_REPORT.md report template (auto-filled per run) MUNYARWANDA_BENCH.md evaluation suite documentation ``` ## Quickstart (free Google Colab T4) 1. Open `notebooks/Munyarwanda-AI-v0.3 …