YinkaAjao/African-Language-Health-QA-Challenge

Multilingual Health Question Answering in Low-Resource African Languages: A Deep Learning Approach # African-Language-Health-QA-Challenge Multilingual Health Question Answering in Low-Resource African Languages: A Deep Learning Approach ## Project Overview Developing healthcare Artificial Intelligence (AI) for low-resource African languages presents a critical dual challenge: the catastrophic risk of medical hallucinations and the severe scarcity of high-quality training data. This repository contains the complete end-to-end Machine Learning pipeline I engineered to answer health queries in dialects such as Swahili, Akan, Luganda, and Amharic. Through a rigorous progression of 27 experiments, I mapped the theoretical limits of statistical lexical algorithms (TF-IDF, BM25), parameter-efficient generative fine-tuning (mT5, ByT5, Gemma-2B), and dense semantic retrieval (E5, BGE-M3). **Peak Performance:** Achieved a **Zindi Score of 0.5718** using a State-of-the-Art Dense Bi-Encoder (`BAAI/bge-m3`) augmented with strict algorithmic language-constraint masks. --- ## Key Findings & Architectural Breakthroughs Throughout this research, I discovered that industry-standard NLP techniques built for high-resource languages actively degrade performance when applied to morphologically complex, data-starved African languages. 1. **The Generative Bottleneck:** Small encoder-decoder models (~300M parameters) lack the capacity to learn agglutinative African grammar from subsampled data. Attempting Retrieval-Augmented Generation (RAG) on zero-shot LLMs caused catastrophic syntax failure. 2. **The Agglutinative Trap:** Upgrading traditional lexical models to BM25 crashed accuracy scores because whitespace tokenization destroys shared morphological prefixes/suffixes. Character-level N-Grams (TF-IDF) are far superior for African textual roots. 3. **The Cross-Lingual Semantic Flaw:** Dense embedding models (like `E5-Small`) perfectly map universal medical concepts but completely ignore dialect barriers, confidently returning perfect medical advice in the wrong language (s …