chaima-Khenafif03/qwen-darija-domain-adaptation

QLoRA continued pretraining of Qwen2.5-3B-Instruct on Algerian Darija text — stage 1 of a meeting-minutes generation pipeline # Qwen2.5 Darija Domain Adaptation QLoRA continued-pretraining of Qwen2.5-3B-Instruct on Algerian Darija text (Latin and Arabic script, code-switched with French) — the first of a two-stage pipeline built for automated meeting-minutes generation on code-switched Algerian workplace speech, during a master's internship at Djezzy. ## Why this approach There's no clustering- or generation-ready LLM with strong native Darija understanding, so rather than relying on translation or a smaller task-specific model, this stage adapts a general-purpose LLM directly to the dialect via continued pretraining, before a second stage fine-tunes it on 500 code-switched meeting transcripts for the actual minutes-generation task. This end-to-end approach outperformed a sequential translate-then-generate pipeline across every metric measured (ROUGE-L +29.7 pts, BERTScore F1 +3.0 pts, schema compliance +4.1 pts). ## Approach - **Base model**: `Qwen/Qwen2.5-3B-Instruct`, loaded in 4-bit (QLoRA: nf4 quantization, double quant, fp16 compute) - **LoRA**: rank 16, alpha 32, applied to all attention and MLP projections (`q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`) - **Objective**: causal language modeling (continued pretraining), not instruction tuning - **Data**: a 182K-sample corpus of Algerian social media text; **this run used a random 10K subset**, due to Kaggle's free-tier GPU quota — a real hardware constraint, not an oversight. The code supports the full corpus if more compute is available. - 1 epoch, batch size 2 with gradient accumulation 8 (effective batch 16), learning rate 2e-4 ## Results Best validation perplexity: **25.78** ## Project structure ``` ├── src/ │ ├── preprocess.py # data loading and tokenization │ ├── train.py # QLoRA training loop │ └── predict.py # generation from the adapted model ├── data/ │ └── sample.txt # sample of the domain adaptation corpus (full 182K corpus not published) └── requ …