BanHealthAIRespRelv3: A Benchmark Dataset for Evaluating Bengali AI-Generated Health Responses Based on User-Specific Prompts

BanHealthAIRespRelv3 serves as a benchmark dataset for evaluating the relevance and safety of AI-generated health responses in Bengali, a low-resource language. Its primary goal is to address risks of medical hallucinations, irrelevant advice, and misinformation in online health information seeking (OHIS), particularly in resource-constrained settings like Bangladesh. To ensure high quality, the dataset underwent rigorous preprocessing (text cleaning, duplicate removal, filtering incomplete responses) and annotation by three native Bengali speakers, with reliability confirmed via Fleiss’ Kappa (substantial to almost perfect agreement) and retention of only high-confidence instances (>0.8). BanHealthAIRespRelv3 is a ternary-class dataset comprising 15,384 instances of user-simulated health prompts and corresponding AI responses (from ChatGPT and Google Gemini), categorized as: Highly Relevant: 5,004 instances Partially Relevant: 5,300 instances Not Relevant: 5,084 instances The dataset has broad applications in multiple NLP and AI areas, including: Relevance and hallucination detection in health AI Development of safer, culturally sensitive health chatbots Trust calibration and ethical AI in digital health Misinformation mitigation in low-resource languages Educational and research advancements in Bengali medical NLP BanHealthAIRespRelv3 is openly available for academic and research purposes, promoting collaboration and innovation in Bengali NLP. By establishing a robust benchmark, it aims to foster trustworthy, context-aware AI systems for health-related applications in under-resourced languages.