Salman-Elhaddadi/darija-dialect-classifier

Binary MSA-vs-Darija dialect classifier: from-scratch PyTorch NN vs a logistic regression baseline, under a disciplined eval protocol. # Darija Dialect Classifier Binary MSA-vs-Darija dialect identification on Arabic-script tweets: a TF-IDF + logistic regression baseline, and a from-scratch feedforward neural net built to test whether it can beat that baseline — under a disciplined split/eval protocol, not just a bigger-model guess. **Result: the NN didn't beat the baseline (0.843 ± 0.005 vs. 0.862 test macro-F1), and the reason why is diagnosed, not hand-waved. Full write-up in `RESULTS.md`.** ## Task Given a tweet, classify it as **standard** (Modern Standard Arabic) or **dialectal** (Darija, Moroccan Arabic). Originally scoped as a sentiment classifier — the corpus does carry sentiment labels (positive/neutral/negative/mixed) — but it turned out to be ~70% MSA rather than Darija, so a Darija-specific sentiment task would mean filtering down to genuinely-dialectal rows first; after dedup that subset was too small (~3.6k rows across 4 uneven sentiment classes) to be workable. Narrowed to dialect ID instead, where the full corpus is usable. ## Data MAC corpus — not vendored in this repo (the source states no license), fetched at build time instead. After cleaning (URL/mention stripping, diacritic/tatweel removal, dedup on normalized text) and a stratified 70/15/15 split: 8,642 train / 1,852 val / 1,852 test, ~29% dialectal in each. ## Approach Both models share the same TF-IDF-over-character-n-gram *encoding scheme* (char_wb 2-5, sublinear TF-IDF, L2-normalized) — but not the same feature space: the NN prunes its vocabulary harder (`min_df=10` vs. the baseline's `min_df=2`, 13,194 features vs. 51,185) to keep its first-layer parameter count in line with the 8,642-row training set. So the comparison isolates the architecture change plus that deliberate, capacity-driven feature reduction — not architecture alone. Every neural-net design decision — input representation, vocab handling, encoding, depth/width, output layer + loss, regularization, split discipline, optimizer/hyperparameters, eval …