Jonathan-321/kilm

Standalone Kinyarwanda language-model sandbox # KILM KILM is a standalone learning and feasibility sandbox for the from-scratch Kinyarwanda language-model track. It is separate from the main `kinyalm` planning repo, separate from the CS336 assignment repos, and separate from the final training path. The question this sandbox answers is: ```text Can we run the whole Track A loop end to end before real data and larger models? ``` The loop is: ```text text corpus → tokenizer → token IDs → training batches → tiny causal language model → loss curve → sample generation → written interpretation ``` ## Production-Scale Baseline Path The repo now includes a larger from-scratch Kinyarwanda LM path in addition to the original tiny sandbox loop: ```bash make aggregate make train-tokenizer make tokenize-full make train-full make final-report ``` That path aggregates the large Kinyarwanda corpus, trains a 32k SentencePiece BPE tokenizer with byte fallback, tokenizes into 1024-token Arrow blocks, and trains a 109M-parameter LLaMA-style causal LM from scratch. The training script defaults to a 50000-step target with AdamW, cosine LR, 2000 warmup steps, gradient clipping, checkpointing, validation, and sample generation every 2000 steps. On local Apple MPS, the recorded baseline completed 2000 steps as a compute-limited run; see `docs/FINAL_RUN_REPORT.md`, `docs/DATA_CARD.md`, and `docs/MODEL_CARD.md`. Generated text is still not fluent enough for product use. Treat the current model as proof that the pipeline works, not proof that the model is ready. ## Conversation SFT Pivot For a usable Kinyarwanda assistant, the next path is supervised fine-tuning on real user/assistant conversations, then human-rated evaluation. Prepare conversation data: ```bash python scripts/prepare_sft_conversations.py \ --input data/sft/raw_conversations.jsonl \ --out-dir data/sft/processed \ --validation-fraction 0.1 ``` Bootstrap translation-style SFT pairs from open Hugging Face datasets: ```bash python scripts/generate_sft_bootstra …