Christabll/meditron-ethiopia-filtering

3-signal filtering pipeline (heuristics + translation-quality + LLM-as-judge) to test whether an LLM can evaluate Amharic medical translation quality — MeditronFO-Ethiopia. # OSS filtering — README Filtering the OSS (MedQA) Amharic medical answers with a 3-signal filter, to test whether an LLM can judge Amharic translation quality or whether we need native speakers. ## Why the sample_ids aren't sequential The source pool is the **first 100 OSS samples** (`first_100_oss.jsonl`, ids 1–100). I didn't take ids 1–25 in order — I **selected** a spread of 25 on purpose so the batch would surface errors: - all 3 answers the model got **wrong** - some **hard** ones (model needed many retries) - a mix of **medium** and **easy** ones That's why the 25 have scattered ids (41, 43, 66, 2, 25, …). The **remaining 75** are the other ids from the first 100. ## The 3 signals 1. **Heuristics** — is it really Amharic (script), length ratio, empties. Catches obviously broken output. 2. **Translation quality (LaBSE)** — embedding cosine similarity between the English answer and the Amharic answer (0–1). Higher = same meaning. This is a reference-free, COMET-style check (not literally COMET — that model is gated on HuggingFace). 3. **LLM-as-judge (Gemini)** — Gemini scores each answer on the annotation schema (see `scoring_rubric.md`). ## Files **Data (raw text)** - `first_100_oss.jsonl` — the first 100 OSS samples (source pool) - `oss_selected_25.json` — the 25 I selected - `oss_selected_75.json` — the remaining 75 **Results (the 25)** - `step_a_results.csv` — heuristics - `step_b_llm_scores.csv` — Gemini judge scores - `step_b2_quality.csv` — translation-quality scores - `combined_A_B.csv` — heuristics + Gemini merged - `all_three_signals.csv` — **all 3 signals in one table (start here)** **Independent judge check** - `independent_judge_groq.ipynb` — the same 25 scored by Llama (via Groq), with a Gemini-vs-Llama comparison - `independent_judge_scores.csv` — Llama's scores **For native-speaker scoring** - `native_speaker_sheet.csv` — the 25, blank score columns (model scores left out so they don't bias the human) - `scoring_rubric.md` — the rubri …