Armstrong66/medisafe-gh

The premier medical safety benchmark for chatbots/LLMs in Ghana. # MediSafe-GH - G-MASS **Ghana Medical AI Safety Screen** -- an open-source evaluation pipeline for testing whether medical AI assistants behave safely when queried in English, Twi, and Ghanaian English. Africa AI Safety Prize Competition 2026 - Track II submission. ## What The Pipeline Does G-MASS evaluates model responses with clinical safety metrics, not just general helpfulness or factuality. - **CSR**: Clinical Safety Rate, the percentage of responses scored `SAFE`. - **SDS**: Safety Degradation Score, computed as CSR_English minus CSR_Twi or CSR_GH-EN. - **RAR**: Referral Adequacy Rate, the percentage of crisis-level probes where the model refers the user to a clinician. The active entry point is: ```powershell gmass ... ``` ## Probe-Tested Models The pipeline currently evaluates these 4 probe-tested models: | Key | Model ID | Provider/backend | |---|---|---| | `gpt4o` | `gpt-4o` | OpenAI | | `gemini` | `gemini-2.5-flash` by default, overrideable with `GEMINI_MODEL` | Google GenAI | | `phi3` | `microsoft/Phi-3-mini-4k-instruct` | Local `transformers`, local OpenAI-compatible server, or HF router | | `biomistral` | `BioMistral/BioMistral-7B-SLERP` | Local `transformers`, local OpenAI-compatible server, or HF router | Scorers are not probe-tested models. Do not count these as evaluated models: - `LlamaGuard3` - `AfroLM` - `Gemma` - `meta-llama/Llama-Guard-3-1B` - `google/gemma-3-1b-it` `gemini` can be an evaluated model key. Separately, `SCORER_BACKEND=policy_api` means the scorer pipeline uses a hosted policy-judge runtime to execute scorer prompts. That runtime may currently call Gemini API under the hood, but the scorer identities remain `LlamaGuard3`, `Gemma`, and `AfroLM`. The `gemini` key is a stable pipeline key for a Gemini Flash model, not a claim that the historical `gemini-1.5-flash` endpoint is still available. Current defaults use `gemini-2.5-flash`. For extension notes, see `docs/model-and-scorer-extensibility.md`. ## Environment Se …