SirDamis/yoruba-test-time-scaling

# Yoruba Test-Time Compute Scaling Experiment pipeline for **When Is More Thinking Enough? Evaluation of Test-Time Compute Scaling for Yoruba Language Reasoning**. **Core idea:** Can test-time compute (TTC) scaling compensate for weak low-resource language representations? Evaluation is **Yoruba-only** (AfriMGSM + AfriMMLU test splits). English appears only as an intervention (English CoT, translate-to-English pivot) and via the English-input `*_translate` variants used for baseline comparison. Local vLLM setup: **README_VLLM.md**. ## Research questions | RQ | Question | |----|----------| | **RQ1** | Which reasoning language works best for Yoruba tasks? | | **RQ2** | Does TTC improve performance on low-resource languages like Yoruba? | | **RQ3** | Where is the bottleneck: candidate generation or selection? | | **RQ4** | Can a small model plus TTC match a larger model without TTC? | ## Datasets All runs use **test splits only** (`data/normalized/.../test.jsonl`): | Name | Task | Rows | Path | |------|------|------|------| | `afrimgsm` | Math (Yoruba) | 250 | `math-reasoning/afrimgsm/test.jsonl` | | `afrimmlu` | QA (Yoruba) | 500 | `question-answering/afrimmlu/test.jsonl` | | `afrimgsm_translate` | Math (English input, baseline) | 250 | `math-reasoning/afrimgsm_translate/test.jsonl` | | `afrimmlu_translate` | QA (English input, baseline) | 500 | `question-answering/afrimmlu_translate/test.jsonl` | Normalized JSONL rows: `{"answer_type": "choice|number", "choices": [...] | null, "gold_answer": "...", "question": "..."}` ```bash uv run python scripts/download_hf_datasets.py --dataset all ``` ## Setup ```bash uv lock && uv venv uv pip install -r requirements.txt # cloud GPU image: install CUDA torch first, then vllm if needed ``` Set `HF_TOKEN` for gated models. Qwen3 native thinking is disabled in both backends (`enable_thinking=False`) — this is a prompted-CoT experiment. ## Experiment plan Run order: **E0 → E1 → E2 → E3 → E4**. After E1, set the E2 co …