LotfiDjebbar/Darija-Transformer

From-scratch Transformer encoder (PyTorch) for Algerian Darija sentiment classification : 79.42% accuracy, 27× fewer parameters than a pretrained baseline. # Darija Transformer — Sentiment Classification for Algerian Dialect A Transformer encoder implemented **from scratch** (PyTorch, no `nn.MultiheadAttention`, no `nn.Transformer`) for sentiment classification on Algerian Darija comments — a low-resource language in NLP, characterized by constant code-switching between Arabic script, French, and arabizi (chat-alphabet) within the same message. Portfolio project by **Djebbar Lotfi**, final-year Data Science & AI engineering student, ENP Algiers. *Version française : README.fr.md* · *Full report (PDF): report/rapport.pdf* ## Key results | Model | Parameters | Training time | Accuracy (test) | F1 macro (test) | |---|---|---|---|---| | **From-scratch Transformer (this project)** | 4.65M | ~2 min | **79.42%** | **0.7940** | | Fine-tuned DziriBERT (`alger-ia/dziribert`) | 124.4M | ~12.5 min | 87.39% | 0.8736 | | SVM (paper baseline, raw dataset) | — | — | 68.07% | not reported | | CNN (paper baseline, raw dataset) | — | — | 62.96% | not reported | | RNN (paper baseline, raw dataset) | — | — | 62.11% | not reported | The from-scratch model clearly outperforms the classical baselines from the source paper, with no pretraining at all, using **27× fewer parameters** than DziriBERT. The remaining gap with DziriBERT is explained by pretraining on 1M+ authentic Algerian tweets, not by a smarter architecture — see Analysis below. *Sentence embeddings (test set) projected to 2D with t-SNE. Errors (orange) concentrate in the overlap zone between the two sentiment clusters rather than being scattered randomly — evidence the model learned a coherent representation space.* ## Why this project Algerian Darija is under-resourced in NLP: no standardized orthography, constant mixing of Arabic/French/arabizi, very few publicly annotated corpora. This project builds the core components of a Transformer encoder by hand (multi-head self-attention, sinusoidal positional encoding, encoder blocks) to demonstrate a fundamental understand …