dsfsi/gov-za-multilingual

The data set contains cabinet statements from the South African government. Data was scraped from the governments website: gov.za # Gov-ZA Multilingual Cabinet Statements **A sentence-aligned multilingual corpus of South African government cabinet statements in 11 official languages** 🤗 Dataset • 📄 Paper • 🗂️ Zenodo • 📊 Explore Data • 📝 Feedback --- ## 📚 Table of Contents - About - Dataset - Quick Start - Using the Dataset - Development - Pipeline Architecture - Alignment Statistics - Citation - License - Contributors --- ## 🌍 About This repository contains **sentence-aligned parallel text** from South African government cabinet statements in **11 official languages**. The data is scraped from gov.za, maintained by the Government Communication and Information System (GCIS). ### Key Features - 🌐 **724,694 aligned sentence pairs** across 55 language combinations - 🔗 **Sentence-level alignment** using LASER embeddings - 📈 **High-quality alignments** with confidence scores (cosine similarity ≥ 0.65) - 🎯 **Ready-to-use splits** (train/test/eval) for machine learning - 🚀 **Automated pipeline** with GitHub Actions for continuous updates - 🤗 **Available on Hugging Face** for easy integration ### Supported Languages | Language | Code | |----------|------| | English | `eng` | | Afrikaans | `afr` | | isiNdebele | `nbl` | | isiXhosa | `xho` | | isiZulu | `zul` | | Sesotho | `sot` | | Language | Code | |----------|------| | Sepedi | `nso` | | Setswana | `tsn` | | Siswati | `ssw` | | Tshivenda | `ven` | | Xitstonga | `tso` | | | | --- ## 📦 Dataset ### Hugging Face Datasets 🤗 The sentence-aligned dataset is available on Hugging Face for easy integration with ML workflows: ```python from datasets import load_dataset # Load a specific language pair dataset = load_dataset("dsfsi/govza-sa-cabinet-statements-sentence-aligned", "afr-eng") # Access splits train_data = dataset["train"] # ~70% of data test_data = dataset["test"] # ~15% of data eval_data = dataset["eval"] # ~15% of data # Iterate through examples for example in train_data: print(f"Afr …