AfriSpeech/africa-mt-builder

Automated HTTP scraper that builds African-language Bible parallel text datasets from YouVersion — 1,090 versions, 792 languages, with English/French/Arabic/Chinese/Portuguese pivots # Africa Machine Translation Parallel Corpus Builder A toolkit for building parallel text datasets from African Bible translations on YouVersion. It pairs each local-language Bible verse with its English equivalent and saves the results as clean CSVs ready for machine-translation training and NLP research. Unlike the per-language `ghana-mt-builder`, this edition is **fully automated**: you give it a CSV of YouVersion versions and it scrapes **every row, one language at a time** — you never type a version code. --- ## What it does YouVersion hosts hundreds of Bible translations across the African continent. Each translation is aligned verse-by-verse against a common reference, which makes Bible text one of the best naturally-occurring sources of parallel sentences for low-resource African languages. This project reads a CSV of versions, scrapes the verse pairs for each one, cleans the text, and saves them as structured CSV files — a collection of sentence-level `(local language, English)` pairs ready to train or fine-tune machine-translation models. It uses bible.com's internal chapter JSON API (one request per chapter, ~30× fewer than per-verse scraping). The English side is batch-cached the first time a chapter is fetched and reused for every subsequent language, so English costs near-zero extra requests across the whole run. **No Chrome / Selenium required.** ## Quick start ### Requirements - Python 3.10 or later > All Python dependencies are installed automatically on first run. You do not > need to run `pip install` yourself. ### Clone and run ```bash git clone github.com cd africa-mt-builder # Scrape every version listed in the default CSV: python youversion_parallel_text_builder.py # Or point it at your own CSV: python youversion_parallel_text_builder.py path/to/my_versions.csv ``` The script will: 1. Install any required packages in the background. 2. Load every viable row from the CSV. 3. Scrape …