teganmosibineba/multimodal-healthcare-uncertainty-audit
Domain:
healthcarenatural language processingRecord type:
project
Creator:
teg
Host:
Evaluating Uncertainty in Multimodal Agents for African Healthcare: A Baseline for Policy-Collapse Auditing
# multimodal-healthcare-uncertainty-audit
Evaluating Uncertainty in Multimodal Agents for African Healthcare: A Baseline for Policy-Collapse Auditing
# Multimodal Healthcare Uncertainty Audit
## A 400-Evaluation Baseline Using AfriMed-QA, SLAKE, and PathVQA
This repository contains a reproducible baseline audit of confidence, calibration, abstention, and visual robustness in LLaVA-1.5-7B across African clinical questions, radiology questions, and pathology questions.
The baseline was developed as the first empirical stage of a broader project titled:
> **Evaluating Policy Collapse and Un**rtainty Quantification of Multimodal Agents in African Healthcare Settings**
The current experiments do not claim to demonstrate policy collapse. Instead, they establish the pre-adaptation measurements required to investigate whether later reward-based training introduces behavioural instability, confidence deterioration, answer concentration, evidence insensitivity, or policy collapse.
---
## Public Summary
How well does AI know when it is wrong?
This project investigates whether multimodal AI systems can recognize when their healthcare answers may be unreliable. It measures confidence, calibration, abstention, and sensitivity to degraded medical images and clinical questions relevant to African contexts.
---
## Research Motivation
Multimodal vision-language models are increasingly capable of processing medical images and clinical questions. However, evaluation based on accuracy alone cannot determine whether a model:
- knows when it is likely to be wrong;
- becomes less confident when evidence quality deteriorates;
- abstains when the available information is insufficient;
- remains stable across altered input conditions;
- or produces incorrect answers with unjustified confidence.
These issues are particularly important in healthcare, where a plausible but incorrect response may be difficult for a non-specialist user to identify.
Medical AI benchmarks are also une …
Similar
Multimodal in situ rice leaf physiology dataset for uncertainty-aware adaptive sampling and measurement-quality assessmentAn audit of healthcare provision in internally displaced population camps in NigeriaMultimodal medical diagnosis: a mini review of LLM–vision fusion models in low-resource healthcare settingsA large scale multimodal dataset for healthcare domain Ghanaian sign language translation and retrieval based synthesisForensic Audit Technology and Audit Report Quality of Selected Audit Firms in NigeriaAudit Planning and The Reliability of Audit Evidence: Evidence from External Audit Firms in Ghana