HafsatiMohammed/darija-lid-benchmark

Reproducible pipeline for building a balanced Darija language identification benchmark with Darija Arabic, Darija Latin, and non-Darija examples. # Darija LID Benchmark This repository builds a Moroccan Darija language identification dataset from multiple public sources, normalizes them into one schema, removes duplicates, validates the final output, and supports publication to Hugging Face. ## Goal The target task is 3-way text classification: - `0` = `arabic_moroccan` - `1` = `arabizi_moroccan` - `2` = `other` The `other` class includes non-Moroccan dialects, English translation rows, and the remaining non-target content. ## Final Dataset Schema The final dataset contains these columns: - `text`: normalized text - `dialect`: normalized dialect name - `src`: source dataset identifier, or multiple identifiers joined by `|` - `has_latin_script`: whether the row contains at least one Latin character - `label`: numeric class ID Deduplication is done on normalized `text`. ## Source Datasets Configured in configs/dataset.yaml: - `atlasia/Darija_LID_Anootation_10k` - `atlasia/DODa-audio-dataset` - `atlasia/Darija-LID` - `UBC-NLP/alexandria` - `atlasia/levantine_dialects` ## Pipeline The pipeline has four stages: 1. `scripts/download_data.py` - Downloads the configured raw datasets into `data/raw/` 2. `scripts/prepare_sources.py` - Flattens each source into the shared schema - Maps source-specific labels to the final classes - Preserves source provenance 3. `scripts/build_dataset.py` - Merges prepared sources - Removes repeated rows - Drops conflict groups where the same normalized text has inconsistent labels or dialects - Writes the final dataset to `data/processed/` 4. `scripts/validate_dataset.py` - Checks schema, label validity, duplicate removal, and script-flag consistency ## Run Install dependencies: ```bash pip install -r requirements.txt ``` Run the full data pipeline: ```bash make data ``` Run validation: ```bash make validate ``` Or run the steps manually: ```bash python scripts/download_data.py python scripts/prepare_sources.py python scripts/build_dataset.py python scripts/va …