IlyasFardaouix/darija-dataset-builder

Scalable pipeline for building Moroccan Darija NLP datasets for LLM training. # darija-dataset-builder **Multi-source pipeline for building large-scale Moroccan Darija NLP datasets ready for LLM fine-tuning** ## Motivation Moroccan Darija is spoken by more than 30 million people, yet it remains significantly underrepresented in modern NLP research and open datasets. Most public resources prioritize Modern Standard Arabic or high-resource languages, leaving Darija practitioners with limited benchmark-quality corpora for training and evaluation. Existing Arabic NLP datasets also miss core Darija properties, including frequent code-switching and lexical mixing across Arabic, French, Amazigh, and Spanish influences. This linguistic reality makes off-the-shelf Arabic pipelines unreliable for many Moroccan real-world use cases. `darija-dataset-builder` is designed to close this gap through a scalable, open-source, community-driven pipeline. It focuses on quality-first collection, normalization, deduplication, and export so datasets can be used directly for LLM fine-tuning, adaptation, and evaluation. ## Pipeline Architecture ```mermaid flowchart LR A[Data Sources CommonCrawl, Twitter/X, YouTube comments, existing Darija datasets, web scraping] B[Raw Collection] C[Language Detection (filter for Darija)] D[Text Cleaning (remove HTML, normalize chars, fix encoding)] E[Deduplication (MinHash / exact match)] F[Quality Filtering (length, perplexity score)] G[Output HuggingFace Dataset format] A --> B --> C --> D --> E --> F --> G ``` ## Supported Data Sources | Source | Type | Language | Approximate Size | |---|---|---|---| | CommonCrawl MA subset | Web pages | Darija + code-switched | 8.5M lines | | Twitter/X public stream | Social posts | Darija/Arabic/French mix | 3.2M posts | | YouTube comments dump | User comments | Darija-heavy informal text | 5.1M comments | | Open Darija corpora packs | Public datasets | Curated Darija samples | 1.4M lines | | Targeted Moroccan forums scrape | Forum threads | Darija + Arabic variants | 2.0M lines …