intron-innovation/AfriMed-QA

African Medical QA Dataset # AfriMed-QA: A pan-African Medical QA Dataset Shield: [![CC BY-NC-SA 4.0][cc-by-nc-sa-shield]][cc-by-nc-sa] This work is licensed under a [Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License][cc-by-nc-sa]. [![CC BY-NC-SA 4.0][cc-by-nc-sa-image]][cc-by-nc-sa] [cc-by-nc-sa]: creativecommons.org [cc-by-nc-sa-image]: licensebuttons.net [cc-by-nc-sa-shield]: img.shields.io Collaborating Organizations: Intron Health, SisonkeBiotik, BioRAMP, Georgia Institute of Technology, MasakhaneNLP, Google Research Funded by: Google Research, Bill & Melinda Gates Foundation, PATH, #### Summary AfriMed-QA creates a novel large foundational open-source dataset of 24,000 pan-African clinically diverse questions and answers to rigorously evaluate LLMs for accuracy, factuality, hallucination, demographic bias, potential for harm, comprehension, and recall. The dataset’s geographical and clinical diversity facilitates robust contextual evaluation of LLMs in African healthcare and provides a sufficiently large corpus to finetune LLMs to mitigate biases discovered. #### Background The meteoric rise of LLMs and their rapid adoption in healthcare has driven healthcare leaders in developed economies to design and implement robust evaluation studies to better understand their strengths and weaknesses. Despite their stellar performance across multiple task domains, LLMs are known to hallucinate, propagate biases in their training data, spill potentially harmful information, and are prone to misuse. Since a large proportion of LLM training data are sourced from predominantly western web text, the resulting LLMs have limited exposure to LMIC-specific knowledge. Furthermore, there is minimal evidence to show that stellar performance touted in western literature transfers to healthcare practice in developing countries given their underlyi …