ismailSadouki/dziri-dpo

End-to-end DPO alignment pipeline from scratch. Implements DPO mechanics in pure PyTorch with numerical verification against TRL, followed by SFT and DPO training, preference data construction, annotation agreement, and evaluation for Algerian Darija alignment. # DziriDPO **End-to-end DPO alignment pipeline from scratch.** Implements DPO mechanics in pure PyTorch with numerical verification against TRL, followed by SFT and DPO training, preference data construction, annotation agreement, and evaluation for Algerian Darija alignment. > **Public framing:** verified DPO mechanics + Darija alignment methodology — not simply “I fine-tuned a model.” --- ## Overview DziriDPO is a two-stage project for understanding and building Direct Preference Optimization (DPO) systems. The project separates **DPO mechanics verification** from **production Darija alignment**: ```text DziriDPO │ ┌────────────┴────────────┐ │ │ ▼ ▼ Scratch DPO Darija Alignment Pure PyTorch TRL │ │ ▼ ▼ DPO mechanics Data + SFT + DPO │ │ ▼ ▼ TRL numerical oracle Triple-axis evaluation │ │ └────────────┬────────────┘ ▼ Reproducible results ``` The central correctness oracle is: $$ |L_{\text{scratch}} - L_{\text{TRL}}| **What makes the chosen response better than the rejected response in Algerian Darija?** The guideline considers properties such as: * instruction following * factual correctness * relevance * clarity * natural Darija usage * appropriate code-switching * cultural/contextual appropriateness * harmful or misleading content The detailed guideline is maintained in: ```text darija_alignment/data/guideline.md ``` --- # Inter-Annotator Agreement A subset of at least **100 preference pairs** is independently double-annotated. Cohen's kappa is computed as: $$ \kappa ====== \frac{p_o-p_e}{1-p_e} $$ where: * $p_o$ is observed agreement * $p_e$ is expected agreement by chance The report includes: * number of double-annotated pairs * observed agreement * expected agreement * Cohen's $\kappa$ * disagreement categories * re …