baggie11/Multilingual-Health-QA-LoRA-Fine-Tuning

Healthcare QA quality is often weaker in low-resource languages. This project fine-tunes an African-language base model on language-specific subsets (for example Swa_Ken) using supervised instruction tuning from (input -> output) examples. # Multilingual Health QA (LoRA Fine-Tuning) A reproducible training pipeline for multilingual health question answering using Unsloth, 4-bit quantization, and LoRA adapters. ## Overview Healthcare QA quality is often weaker in low-resource languages. This project fine-tunes an African-language base model on language-specific subsets (for example `Swa_Ken`) using supervised instruction tuning from `(input -> output)` examples. ## Key Features - Subset-specific training via `subset` filtering - Alpaca-style instruction formatting for SFT - Parameter-efficient LoRA fine-tuning - 4-bit model loading for lower GPU memory usage - Step-wise validation with best-checkpoint selection by `eval_loss` ## Approach 1. Load `Train.csv` and `Val.csv` 2. Filter rows by target `subset` 3. Format each sample into instruction/input/response prompt text 4. Build Hugging Face datasets 5. Load base model in 4-bit 6. Attach LoRA adapters to attention/MLP projection layers 7. Train with `trl.SFTTrainer` 8. Save LoRA adapter and tokenizer ## Default Training Configuration - Base model: `vutuka/Llama-3.1-8B-african-aya` - Sequence length: `1024` - LoRA rank/alpha/dropout: `16 / 16 / 0.0` - Epochs: `2` - Per-device train batch size: `2` - Gradient accumulation: `4` (effective batch size `8`) - Learning rate: `2e-4` - Optimizer: `adamw_8bit` - Scheduler: `cosine` ## Repository Structure - `src/mhqa/train.py`: training CLI - `src/mhqa/infer.py`: inference CLI - `src/mhqa/data.py`: data loading and prompt formatting - `src/mhqa/config.py`: default training config and language map - `scripts/train_swahili.sh`: example training command - `configs/train_swahili.example.yaml`: sample settings file - `data/`: dataset location (`Train.csv`, `Val.csv`) ## Setup ```bash python -m venv .venv # Linux/macOS source .venv/bin/activate # Windows PowerShell # .venv\Scripts\Activate.ps1 pip install --upgrade pip pip install -e . ``` Colab-compatible dependency pinning used in experiments: ```bash …