samzana/multilingual-health-qa-africa

Experiments for a multilingual health QA task across 8 language subsets in 4 African countries: retrieval, NLLB fine-tuning, and hybrid pipelines. # Multilingual Health QA — Zindi Competition Experiments for the Zindi Multilingual Health QA competition. The task: given a health question in one of 8 language subsets across 4 African countries, generate a correct and fluent answer. **Best leaderboard score: 0.70** (top score on board: 0.76) --- ## Competition Overview | | | |---|---| | **Languages** | English (×4 countries), Akan/Twi, Amharic, Luganda, Swahili | | **Countries** | Ethiopia, Ghana, Kenya, Uganda | | **Subsets** | `Eng_Eth`, `Eng_Gha`, `Eng_Ken`, `Eng_Uga`, `Aka_Gha`, `Amh_Eth`, `Lug_Uga`, `Swa_Ken` | | **Train / Val / Test** | ~30 k / ~3 k / ~2.6 k | | **Metric** | ROUGE-1 × 0.37 + ROUGE-L × 0.37 + LLM Judge × 0.26 | --- ## Approaches & Results Experiments ran in roughly this order. Each builds on findings from the previous. | # | Approach | Val ROUGE-1 | Leaderboard | |---|----------|-------------|-------------| | 1 | mT5-small + LoRA | ~0.27 | — | | 2 | mT5-small + LoRA v2 (fixed tokenisation) | ~0.30 | — | | 3 | NLLB-200-600M + QLoRA + oversampling | ~0.64 | 0.485 | | 4 | NLLB-200-1.3B + LoRA + oversampling | ~0.34 | 0.407 | | 5 | Ensemble (600M ROUGE cols + 1.3B LLM col) | — | ~0.50 | | 6 | Few-shot Qwen2.5-7B (no training) | — | — | | 7 | Dense retrieval — E5-large (English only) | — | — | | 8 | Dense retrieval — multilingual E5-large (all languages) | ~0.70 | **0.70** | | 9 | Retrieval + cross-encoder reranking (BGE, MS-MARCO) | — | — | | 10 | QA-quality filtering + Qwen3-0.6B hybrid | — | — | | 11 | Retrieval + answer editing (Qwen3-0.6B) | — | — | | 12 | Improved retrieval (BM25 hybrid, per-subset strategy) | — | — | **Key finding**: pure dense retrieval with `multilingual-e5-large` matched or outperformed all fine-tuned generation models on this dataset, because many test questions are semantically near-identical to training questions. --- ## Repository Structure ``` . ├── utils.py # Shared utilities (ROUGE, embeddings, retrieval, Qwen3 generation) ├── …