michsethowusu/ghana-farmer-qa

# Ghana Farmer QA A pipeline that turns **YouTube videos of Ghanaian farmers talking about their work** into **published parallel question-answer datasets** in English, Ewe, Ga and Twi. The point is grounding: every QA pair traces back to something a farmer actually said on camera, and the source passage travels with it through every stage, so a published answer can always be checked against the speech it came from. ``` YouTube URLs │ 01 download audio, split into 30s WAV chunks ▼ audio_chunks/*.wav │ 02 speech recognition (Twi) ▼ metadata.csv file, transcription │ 03 Gemini translation ▼ english_translations.csv file_name, original_twi, translated_english │ └── 04, 05 (optional) publish the speech + parallel corpora │ 06 Gemini QA generation, ~10 standalone pairs per passage ▼ farmer_qa_output.jsonl nested: passage → categories → QA pairs │ 07 flatten ▼ farmer_qa_flattened.csv one row per QA pair, + source passage │ 08 translate into Ewe / Ga / Twi ▼ farmer_qa_translated.csv │ 10 validate ── fails the run if translations are empty, │ unchanged, wrong-language, or hold pivot residue ▼ farmer_qa_clean.csv │ 09 split per language, write cards, push ▼ Hugging Face datasets ``` ## Quick start ```bash pip install -r requirements.txt # also needs ffmpeg on PATH export GEMINI_API_KEY=... # stages 3 and 6 hf auth login # stages 4, 5, 9 ./run_pipeline.sh # every stage ./run_pipeline.sh 6 10 # only stages 6-10 ORGS=myorg ./run_pipeline.sh 9 9 # publish only ``` Every stage checkpoints and resumes, so re-running after an interrupt picks up where it stopped rather than redoing work. Each script also runs standalone with `--help`. ## Stages | # | Script | Does | Key inputs → output | |---|---|---|---| | 1 | `01_download_youtube_audio.py` | Downloads audio, converts to 16 kHz mono, sl …