BouajilaHamza/Tunisia-msa-parallel-corpus

### Dataset Description This is an ambitious project to create a high-quality, reproducible parallel corpus for Modern Standard Arabic (MSA) and Tunisian Arabic (aeb) through a sophisticated synthetic data generation pipeline. The dataset is being developed by the Tunisia.AI community to address the scarcity of high-quality dialectal data for training and evaluating language models. The primary goal is to provide a rich, well-documented resource for the research and development of: - Machine translation systems between MSA and Tunisian Arabic. - Dialectal-aware text generation models. - Cross-dialectal understanding and representation learning. This dataset is created with an emphasis on **rigorous quality control**, **reproducibility**, and **ethical considerations**. ### Dataset Status This is an early-stage release (`v0.1.0`), and this dataset card serves as a public blueprint for the project. The final dataset is currently being generated. This repository is intended for development and will be populated with the full data, scripts, and evaluation artifacts upon completion. ### Dataset Structure The dataset will be provided in `JSONL` format, with each row corresponding to a single parallel sentence pair. The schema includes not only the source and target text but also rich metadata for provenance and quality control. | Column | Type | Description | |---|---|---| | `id` | `string` | Unique example id (UUID). | | `source` | `string` | Final MSA sentence (used as training input). | | `target` | `string` | Tunisian sentence (used as training target). | | `source_dialect` | `string` | The original raw Tunisian text (for provenance). | | `msa_generated` | `string` | The initial MSA candidate before any edits. | | `score_composite` | `float` | The composite quality score (0-1) used for acceptance. | | `cosine_similarity` | `float` | Semantic similarity between Tunisian and MSA embeddings. | | `lm_logprob` | `float` | Fluency score from an MSA Language Model. | …