smutuvi/ndizi-mlops-gemma4-trl

TRL + HuggingFace MLOps pipeline for chat-safe Gemma 4 Swahili ASR on NDIZI # ndizi-mlops-gemma4-trl Modular TRL + HuggingFace pipeline for chat-safe Gemma 4 Swahili ASR fine-tuning on `smutuvi/ndizi-1` + `smutuvi/ndizi-1-2025`. Ported from `gemma4_ndizi_asr_trl_hf.ipynb`. ## Layout ``` ndizi-mlops-gemma4-trl/ ├── config_files/ # JSON run configs ├── bash_scripts/ # Shell wrappers ├── scripts/ # Thin CLI entrypoints ├── src/ │ ├── data/ # NDIZI merge + chat replay (Section 5) │ ├── eval/ # WER/CER + chat gate (Sections 8–9) │ ├── inference/ # gemma_build_asr_inputs, transcribe (Section 4b) │ ├── models/ # Load base, LoRA, checkpoint (Sections 3–4) │ ├── pipeline/ # CLI orchestration │ ├── publish/ # Save / merge / Hub push (Section 10) │ ├── training/ # Collator + SFTTrainer (Sections 6–7) │ └── utils/ # Constants, paths, runtime, env ├── artifacts/ # Checkpoints, predictions (gitignored) └── run_pipeline.py ``` ## Setup ```bash cd ndizi-mlops-gemma4-trl pip install -r requirements.txt export HF_TOKEN=... # or add to .env ``` ## Quick start ```bash # Section 4b — sanity on base model (optional --audio path.wav) python run_pipeline.py sanity --config config_files/gemma4_e2b_asr_moderate.json # Section 7 — train (default: asr_moderate, batch=1, 3 epochs, 10% chat replay) python run_pipeline.py train --config config_files/gemma4_e2b_asr_moderate.json # Section 8 — WER/CER eval python run_pipeline.py evaluate --checkpoint artifacts/checkpoints/latest # Section 9 — chat quality gate python run_pipeline.py chat-gate --checkpoint artifacts/checkpoints/latest # Inference on file or directory of .wav python run_pipeline.py infer --checkpoint artifacts/checkpoints/latest --audio /path/to/audio.wav # Publish adapter or merged weights python run_pipeline.py publish --checkpoint artifacts/checkpoints/latest --push ``` Full pipeline: ```bash bash bash_scripts/run_gemma4_trl_all.sh ``` Eval-only …