haakso/luganda-mt-synthetic

# luganda-mt-synthetic Luganda is an extremely low-resource language with limited parallel corpora, making neural machine translation challenging for standard approaches. This project investigates whether synthetic data augmentation can meaningfully improve LG→EN translation quality when combined with parameter-efficient fine-tuning. We benchmark four 7–8B parameter LLMs (Gemma-7B, Mistral-7B, Llama-3.1-8B, Aya-23-8B) zero-shot on a 2,000-pair test set drawn from three corpora, scoring with COMET (wmt22-comet-da). Zero-shot performance is modest across all models (COMET 0.325–0.356), with no single model dominant. We then fine-tune the Aya-23-8B base model with QLoRA on 1× and 5× synthetic training sets, finding that even the 1× condition yields a large gain (COMET 0.544, +0.219) and the 5× condition brings the model to 0.611 (+0.286 over zero-shot), with consistent improvements across all three source corpora. ## Architecture ### Overall pipeline ### QLoRA fine-tuning --- ## Prerequisites - uv for dependency management (host-side download script + local dev) - Docker with nvidia-container-toolkit for evaluation and fine-tuning runs - A HuggingFace account with the Llama 3.1 license accepted ## Codebase map | Path | Description | |---|---| | `scripts/evaluate.py` | Main evaluation harness: loads a candidate model in 4-bit, runs batched greedy inference on the test set, scores with COMET, and logs results to MLFlow | | `scripts/finetune.py` | Sequential QLoRA fine-tuning of Aya 23 8B on the 1x and 5x synthetic conditions; each condition trains, evaluates with COMET, and logs to a nested MLFlow run under a single parent | | `scripts/download_models.py` | Downloads all candidate model weights from HuggingFace into `MODEL_CACHE_DIR`; skips models whose target directory already exists | | `Dockerfile` | CUDA 12.8 + Python 3.11 image managed by uv; bakes in the COMET checkpoint and xlm-roberta encoder at build time so the container is fully air-gapped at runt …