pmwaniki/vignette

Benchmarking Large Language Models and Clinicians Using Locally Generated Primary Healthcare Vignettes in Kenya # Benchmarking Large Language Models and Clinicians Using Locally Generated Primary Healthcare Vignettes in Kenya ## 1. Running LLMs The script `run_llms.py` is used to generate responses from multiple large language models (LLMs) for a set of clinical scenarios. It leverages the LangChain framework to interface with different LLM providers, including OpenAI, Google, HuggingFace, and Ollama. The script reads the scenarios from a CSV file, sends each scenario to the selected LLMs, and stores the responses in a SurrealDB database as well as a CSV file. The models and system prompts can be configured within the script. This process enables standardised benchmarking of LLMs on the same set of clinical queries. ## 2. Descriptive and Ordinal Logistic Regression The data for these analyses are available in the *datasets* folder. 'Combined review data.csv' or 'Combined review data.parquet' contain the expert panel ratings for the 507 vignettes, while 'Prompt responses.xlsx' contains the formatted responses from the clinicians and LLM models. The R scripts `descriptives.R` and `analyses/models.R` provide the statistical analysis pipeline: - **Descriptive Statistics (`descriptives.R`):** This script summarizes the LLM responses by calculating means and standard deviations of scores across different clinical domains and models. It produces summary tables and radar plots to visualize model performance on various dimensions. - **Ordinal Logistic Regression (`analyses/models.R`):** This script fits Bayesian ordinal logistic regression models using the `brms` package. The models compare LLMs on 5-point Likert outcomes across multiple domains, accounting for hierarchical structure (e.g., random effects for panel ). The script includes: - Model fitting with informative and non-informative priors. - Extraction and formatting of fixed effects as odds ratios. - Posterior predictive checks and visualizations. - Pairwise comparisons of models within each domain, reporting differen …