raphaelmerx/openwho-code

OpenWHO: A Document-Level Parallel Corpus for Health Translation in Low-Resource Languages # OpenWHO: A Document-Level Parallel Corpus for Health Translation in Low-Resource Languages 📝 Paper | 📊 Data This repository contains data extraction and evaluation code for the OpenWHO health MT evaluation corpus. ## Getting started 1. Install requirements with `uv sync` 2. Prepare a `.env` file with secrets: ``` OPENROUTER_API_KEY=... HF_TOKEN=... GOOGLE_API_KEY=... ``` ## Data preparation Code for scraping, filtering, and publishing data to the HuggingFace repo is in the `dataprep` directory. ## Evaluation Code for evaluating model perf on OpenWHO and WMT24++ is in the `eval` directory: * `metric_openwho.py`: Script for computing eval metrics (ChrF / MetricX). * `eval_openwho.py`: Script for running inference on NMT models (NLLB / MADLAD) * `eval_gemini_sentlevel.py`: Script for running LLM inference at the sent level (on OpenWHO / WMT24++, with Gemini / Deepseek / Gemma3). * `eval_gemini_sent_window.py`: LLM inference with a sliding context window around the sentence to translate. * `eval_gemini_sent_context.py`: LLM inference with the whole document context, but translating one sentence at a time. * `eval_gemini_doclevel.py`: Script for running LLM inference at the doc level. Argument `--translate-again` is used to translate again.