Abel-Marie/transformers_from_scratch

Full encoder-decoder Transformer built from scratch in PyTorch for Amharic-English translation. # Transformer From Scratch — Amharic ↔ English Translation > Full encoder-decoder Transformer implemented in PyTorch — every layer (multi-head attention, sinusoidal positional encoding, layer norm, feed-forward, residual connections, encoder/decoder stacks, projection head) hand-written, no `transformers.AutoModel`. Trained for Amharic → English neural machine translation. Based on Vaswani et al., *"Attention Is All You Need"* (2017). ## What this does - Implements a complete **encoder-decoder Transformer** from scratch in PyTorch (`src/model.py`). - Trains a **word-level tokenizer** for both source (Amharic) and target (English) using the Hugging Face `tokenizers` library (`src/utils.py`). - Loads parallel Amharic–English text from the **`Helsinki-NLP/opus-100`** dataset (configurable in `config.py`). - Trains the Transformer with **label-smoothed cross-entropy**, **Adam**, and **TensorBoard** logging. - Performs **greedy decoding** for inference (`translate.py`) and end-of-epoch sample translations during training. ## Architecture | Component | Class (`src/model.py`) | |---|---| | Token embeddings (× √d_model) | `InputEmbeddings` | | Sinusoidal positional encoding | `PositionalEncoding` | | Custom layer normalization | `LayerNormalization` | | Position-wise feed-forward (4× d_model) | `FeedForwardBlock` | | Multi-head scaled dot-product attention | `MultiHeadAttentionBlock` | | Pre-norm residual connection | `ResidualConnection` | | Encoder block (self-attn + FF) | `EncoderBlock` | | Stacked encoder | `Encoder` | | Decoder block (masked self-attn + cross-attn + FF) | `DecoderBlock` | | Stacked decoder | `Decoder` | | Output projection to vocab | `ProjectionLayer` | | Full model | `Transformer` | | Factory | `build_transformer()` | ## Default config ```python { "batch_size": 8, "num_epochs": 20, "lr": 1e-4, "seq_len": 350, "d_model": 512, "datasource": "Helsinki-NLP/opus-100", "lang_src": "am", "lang_tgt": "en", } ``` The default architecture is the "base" …