michsethowusu/ghana-chat-data-prep

End-to-end pipeline for preparing and translating Ghana chat corpus into Twi, Ewe, and Ga via Amharic pivot # Ghana Chat Data Prep End-to-end pipeline for preparing and translating the ghananlpcommunity/ghana-chat dataset (226,300 conversations) into Twi (ak), Ewe (ee), and Ga (gaa) using an Amharic pivot via Google Translate. ## Overview ``` ghananlpcommunity/ghana-chat (226,300 conversations) │ ▼ prepare_ghana_chat.py Extract text + generated_question from conversations │ ▼ base.parquet │ ▼ clean_text.py Normalize numbers/dates/units/currency → words │ (also patches existing checkpoints) ▼ base_cleaned.parquet │ ▼ translate_pivot.py EN → Amharic → {Twi, Ewe, Ga} │ (8 stages, checkpointed, auto-cooldown) ▼ translated_full.parquet │ ▼ push_translated.py Push 5 datasets to Hugging Face Hub │ ▼ ghananlpcommunity/ghana-chat-corpus (English, 226,300 rows) ghananlpcommunity/ghana-chat-corpus-ak (Twi, 226,300 rows) ghananlpcommunity/ghana-chat-corpus-ee (Ewe, 226,300 rows) ghananlpcommunity/ghana-chat-corpus-gaa (Ga, 226,300 rows) ghananlpcommunity/ghana-chat-corpus-multilingual (all 4, ~906k rows) ``` ## Scripts ### 1. `prepare_ghana_chat.py` Loads the `ghananlpcommunity/ghana-chat` dataset from Hugging Face and extracts: - `text` — all assistant responses joined by `\n\n` - `generated_question` — first user message - `source_type` / `source` — metadata columns **Input:** `ghananlpcommunity/ghana-chat` (HF dataset) **Output:** `base.parquet` (226,300 rows) ### 2. `clean_text.py` Rule-based text normalization that converts numbers, dates, ordinals, currency (symbols & ISO codes), percentages, and abbreviated units to descriptive word form. Built on num2words and the num2desc package logic. **What it converts:** | Input | Output | |-------|--------| | `$50` | `fifty US dollars` | | `GHS 2 million` | `two million Ghana cedis` | | `January 15, 1990` | `the fifteenth of January in the year nineteen ninety` | | `25%` | `twenty-five percent` | | `5 km` | `five kilometers` | | `1.5 kg` …