lemneya/hassania-qwen-finetune

Fine-tuning Qwen 2.5 for Hassania Arabic dialect with compiled datasets # Hassaniya Qwen Fine-Tuning - HDRP Compliant A fully HDRP-compliant (Hassaniya Dialect Resource Protocol) dataset and pipeline for fine-tuning language models on the Hassaniya Arabic dialect. ## Protocol Compliance Status | Gate | Requirement | Achieved | Status | |------|-------------|----------|--------| | **H1 Corpus Gate** | - | - | ✓ PASSED | | DAPT Tokens | ≥50,000 | 323,900 | ✓ | | SFT Turns | ≥5,000 | 21,336 | ✓ | | Dialogue Ratio | ≥60% | 100% | ✓ | | Leakage | 0 | 0 | ✓ | ## Repository Structure ``` hassania-qwen-finetune/ ├── hdrp/ │ ├── data/ │ │ ├── raw/ # Source data by type │ │ │ ├── whatsapp_exports/ │ │ │ ├── facebook/ │ │ │ ├── websites/ │ │ │ ├── youtube_transcripts/ │ │ │ ├── tv_radio/ │ │ │ ├── parliament/ │ │ │ ├── film/ │ │ │ ├── poetry_azawan/ │ │ │ ├── celebrities/ │ │ │ └── religion/ │ │ └── processed/ │ │ ├── episodes/ # HDRP Episode format │ │ ├── dqs/ # Quality-scored data │ │ ├── exports/ # Train-ready outputs │ │ │ ├── dapt/ # Continued pretraining │ │ │ ├── sft/ # Instruction tuning │ │ │ └── eval/ # Evaluation set │ │ └── manifests/ # Run manifests │ ├── pipeline/ │ │ ├── collector/ # Episode converter │ │ └── refinery/ # Processing pipeline │ ├── specs/ │ │ └── sampling_config.json # Chat-first weights │ └── docs/ │ └── README.md # HDRP documentation └── README.md ``` ## Train-Ready Exports | File | Records | Purpose | |------|---------|---------| | `hdrp/data/processed/exports/dapt/dapt_hassaniya_v2.jsonl` | 6,478 | Continued pretraining | | `hdrp/data/processed/exports/sft/sft_hassaniya_v2.jsonl` | 10,668 | Instruction tuning | | `hdrp/data/processed/exports/eval/eval_hassaniya_v2.jsonl` | 593 | Evaluation | ## Data Formats ### DAP …