lamhaourii/seq2seq-vs-transformer-darija-eng-translation

# Moroccan Darija → English Neural Machine Translation A cross-architectural empirical study comparing three sequence-to-sequence paradigms for low-resource dialectal Arabic translation. --- ## Overview This project trains and evaluates three NMT architectures on a cleaned 59,100-pair Moroccan Darija–English corpus, holding all training conditions fixed to isolate the effect of architectural inductive bias. | Model | Params | BLEU | ChrF | Val PPL | Train ms/batch | |---|---|---|---|---|---| | Baseline BiGRU Seq2Seq | 26.5M | 13.29 | 26.72 | 146.6 | 205.8 | | Attentional Seq2Seq (Bahdanau) | 31.8M | 16.21 | 32.58 | 108.1 | 451.1 | | **Vanilla Transformer** | **11.1M** | **20.78** | **40.67** | **47.6** | **28.7** | --- ## Key Findings - **The Transformer wins on 9/11 criteria** with the fewest parameters — architectural inductive bias beats raw capacity on low-resource data. - **Information bottleneck confirmed**: RNN BLEU drops 29.2% from medium to long sentences (16.18 → 11.45). - **7.2× training speedup**: Transformer at 28.7 ms/batch vs. BiGRU at 205.8 ms/batch on an RTX 2060. - **Attention alignment verified**: Bahdanau heatmaps show near-diagonal alignment for simple sentences and correct cross-diagonal reordering for complex structures. --- ## Dataset Assembled from `atlasia/darija_english` (Hugging Face), combining four sub-corpora: ``` 64,839 raw pairs → 85% Arabic-script density filter (removes Arabizi) → deduplication → BPE length filter = 59,100 pairs → 47,300 train / 5,904 val / 5,896 test ``` Tokenisation: BPE with 12k Darija vocab and 8k English vocab (selected via 5-point sweep). --- ## Architecture Details ### Model 1 — Baseline Seq2Seq 2-layer BiGRU encoder → single fixed context vector → 2-layer GRU decoder. Demonstrates the information bottleneck dilemma. ### Model 2 — Attentional Seq2Seq Same BiGRU encoder with Bahdanau additive attention. Dynamic context vector recomputed at every decoder step. Exports full attention weight …