Logo Lanfrica
  • Home
  • Atlas
  • Insights
  • Docs
  • Sign in

© 2026 Lanfrica. All rights reserved. All copyrights of the resources shown on the Lanfrica website belong to the original copyright holders, unless explicitly stated otherwise.

Evaluating Multimodal LLMs for Inpatient Diagnosis: Real-World Performance, Safety, and Cost Across Ten Frontier Models

Domain:

healthcare

Record type:

paper
Creator:
BasRouMunGra
Host:avatar
Background: Large language models (LLMs) are increasingly proposed for diagnostic support, but few evaluations use real-world multimodal inpatient data, particularly in low and middle-income country (LMIC) public hospitals. Methods: We conducted VALID, a retrospective evaluation of 539 multimodal inpatient cases from a tertiary public hospital in South Africa. Inputs included radiology imaging (CT, MRI, CXR) and reports, laboratory results, clinical notes, and vital signs. Expert panels adjudicated 300 cases (balanced and discordant subsets) to establish ground truth diagnoses, differentials, and reasoning. Ten multimodal LLMs generated zero-shot outputs. A calibrated three-model LLM Jury scored all outputs and routine ward diagnoses across diagnostic accuracy, differential quality, reasoning, and patient safety (>10,000 evaluations). Primary outcomes were composite scores ($S_3$, $S_4$) and win rates. Results: (i) LLM performance was tightly clustered (<15% variation) despite large cost differences; low-cost models performed comparably to top models. (ii) All LLMs significantly outperformed routine ward diagnoses on average diagnostic and safety scores. (iii) Top performance was achieved by GPT-5.1, followed by Gemini models. (vi) Adding radiology reports improved performance by 6%. (v) Diagnostic and reasoning scores were highly correlated ($ρ= 0.85$). (vi) Output rates varied (65-100%) due to input constraints. Results were robust across subsets and evaluation design. Conclusions: Across a real-world LMIC dataset, multimodal LLMs showed similar diagnostic performance despite large cost differences and outperformed routine care on average safety metrics. Affordability, robustness, and deployment constraints may outweigh marginal performance differences in LMIC settings. 17 pages, 11 figures, 10 tables

Visit

arxiv.org

Tags

Machine LearningArtificial Intelligence

Similar

Multimodal large language models versus physicians for real-world inpatient diagnosis: Retrospective comparative (Preprint)Multimodal adaptive cross-mamba for tomato leaf disease diagnosis built on real-world agricultural dataBeyond Metrics: Evaluating LLMs' Effectiveness in Culturally Nuanced, Low-Resource Real-World ScenariosEvaluating Bank Cost Efficiency Using Stochastic Frontier AnalysisAI on the Frontline: Evaluating Large Language Models in Real-World Conflict ResolutionNsanku: Evaluating zero-shot performance of LLMs for Ghanaian languages

Multimodal large language models versus physicians for real-world inpatient diagnosis: Retrospective comparative (Preprint)

BACKGROUND Large language models (LLMs) are increasingly proposed for diagnostic

Multimodal adaptive cross-mamba for tomato leaf disease diagnosis built on real-world agricultural data

Beyond Metrics: Evaluating LLMs' Effectiveness in Culturally Nuanced, Low-Resource Real-World Scenarios

The deployment of Large Language Models (LLMs) in real-world applications presents both opportunitie

Evaluating Bank Cost Efficiency Using Stochastic Frontier Analysis

The study seeks to assess the cost efficiency of the commercial banks in Zimbabwe using the stochast

AI on the Frontline: Evaluating Large Language Models in Real-World Conflict Resolution

This groundbreaking study authored by Nathalie Bussemaker and Mark Freeman and published by the Inst

Nsanku: Evaluating zero-shot performance of LLMs for Ghanaian languages

Nsanku

A project by Ghana NLP to test the machine translation performance of Large Language Models (LLMs) on Ghanaian languages.

Project Overview

Nsanku is an initiative that evaluates how well various open-source language models perform