hoffmann-muki/pidgin-datasets

West African Pidgin datasets for LLM training and Instruction Fine Tuning # Pidgin-English Translation Dataset This repository contains cleaned and processed datasets of English-Pidgin translation pairs and monolingual Pidgin sentences, along with the scripts used to process the data. ## Dataset Overview ### Translation Pairs Dataset - **Language Pair**: English ↔ Nigerian Pidgin (PCM) - **Total Pairs**: 22,001 translation pairs - **Original Source**: OPUS MT560 Dataset - **Hugging Face Dataset**: michsethowusu/english-nigerian-pidgin_sentence-pairs_mt560 - **Source Content**: Religious texts (cleaned of biblical references and religious terminology) - **Format**: Tab-separated pairs ready for machine learning training - **License**: CC-BY-4.0 (Creative Commons Attribution 4.0 International) ### Monolingual Pidgin Dataset - **Language**: Nigerian Pidgin (PCM) only - **Total Sentences**: 176,843 sentences - **Original Source**: AfriBERTa Corpus - **Source Content**: BBC news articles and Common Crawl data - **Format**: One sentence per line, ready for language model pretraining - **License**: Apache-2.0 ## Files ### Translation Pairs Dataset - `pidgin-english-pairs/pidgin-english-pairs.txt` - **Final dataset**: Clean translation pairs in format `pidgin-text\t\tengglish-text` - `pidgin-english-pairs/pidgin-english-data-cleaned.txt` - Intermediate cleaned data with structure - `pidgin-english-pairs/pidgin-english-data.txt` - Initial converted data from parquet ### Monolingual Pidgin Dataset - `pidgin-only/pidgin-sentences.txt` - **Final dataset**: 176,843 Nigerian Pidgin sentences (52.3 MB) - `pidgin-only/train.txt` - Original training split from AfriBERTa corpus (161,843 sentences) - `pidgin-only/eval.txt` - Original evaluation split from AfriBERTa corpus (15,000 sentences) ### Processing Scripts - `pidgin-english-pairs/parquet-to-txt-conversion.py` - Converts parquet files to text format - `pidgin-english-pairs/clean-pidgin-references.py` - Removes biblical references, religious terms, and formatting artifacts - `pidgin-english-pai …