nisbenz/Mini-Transformer

An educational project built from scratch to learn the Transformer architecture by implementing a lightweight English → Darija translator, trained on ~16k sentence pairs. # English → Darija Transformer > it's a hands-on learning exercise to deeply understand how attention, positional encoding, encoder-decoder stacks, and beam search work together. ## Dataset **English-to-Moroccan-Darija** by BounharAbdelaziz on HuggingFace — ~16k parallel sentence pairs. ## References & Inspiration - 📄 Attention Is All You Need — Vaswani et al., 2017 (the original Transformer paper) - 🎥 Coding a Transformer from Scratch — 3Blue1Brown ## Architecture Overview ```mermaid flowchart TB subgraph Input EN["English Sentence"] DA["Darija Sentence (shifted right)"] end subgraph Encoder["Encoder (×3 Layers)"] direction TB EE["Token Embedding + Positional Encoding"] EL1["Encoder Layer 1"] EL2["Encoder Layer 2"] EL3["Encoder Layer 3"] EN_NORM["Final LayerNorm"] EE --> EL1 --> EL2 --> EL3 --> EN_NORM end subgraph Decoder["Decoder (×3 Layers)"] direction TB DE["Token Embedding + Positional Encoding"] DL1["Decoder Layer 1"] DL2["Decoder Layer 2"] DL3["Decoder Layer 3"] DN_NORM["Final LayerNorm"] FC["Linear → Vocab"] DE --> DL1 --> DL2 --> DL3 --> DN_NORM --> FC end EN --> EE DA --> DE EN_NORM -- "Encoder Output" --> DL1 EN_NORM -- "Encoder Output" --> DL2 EN_NORM -- "Encoder Output" --> DL3 FC --> OUT["Output Probabilities"] ``` ## Encoder Layer (Pre-LN) ```mermaid flowchart TB X_IN["Input x"] --> LN1["LayerNorm"] LN1 --> SA["Multi-Head Self-Attention"] SA --> DROP1["Dropout (0.3)"] DROP1 --> ADD1(("+")) X_IN --> ADD1 ADD1 --> LN2["LayerNorm"] LN2 --> FFN["FFN (256 → 512 → 256)"] FFN --> DROP2["Dropout (0.3)"] DROP2 --> ADD2(("+")) ADD1 --> ADD2 ADD2 --> X_OUT["Output"] style ADD1 fill:#4CAF50,color:#fff style ADD2 fill:#4CAF50,color:#fff ``` ## Decoder Layer (Pre-LN) ```mermaid flowchart TB X_IN["Input x"] --> LN1["LayerNorm"] LN1 --> MSA["Masked Self-Attention"] MSA --> DROP1["Dropout (0.3)"] DROP1 --> ADD1(("+")) X_IN --> ADD1 ADD1 --> LN2["LayerNorm"] LN2 --> CA["Cross-Attention (Q=dec, K/V=enc)"] CA --> DROP2["Dropout (0.3)"] DROP2 --> ADD …