nebyu08/sime-more

making unique baby names for ethiopian todlers # sime-more A character-level **GPT** (decoder-only Transformer) implemented **from scratch in pure NumPy** — no PyTorch, no autograd — that learns to generate Ethiopian Geez (ግዕዝ) names. > *sime* = ስም ("name") + makemore vibes ## What it does Trained on ~3,300 Ethiopian Geez names, the model generates plausible new ones: ``` 1. በበድ (novel) 2. አፈ ገበን (novel) 3. አብዱልጀለ (novel) 4. በእሬያሁ (novel) 5. አንግዳር (novel) 6. ኃይለ አለገኝ (novel) 7. ቤተ ድርስ (novel) 8. ገብረ አብ (in train) ... ``` 18 / 20 samples are **novel** (not in the training set), and the model has learned compound-name patterns like `ኃይለ X`, `ገብረ X`, `ቤተ X`, `አፈ X`. ## Architecture A standard pre-norm GPT block, fully hand-implemented: ```diagram ╭─────────────────────╮ token ids ──▶ │ Token Embedding │ ─┐ ╰─────────────────────╯ │ ╭─────────────────────╮ │ add positions ──▶ │ Positional Embedding│ ─┴──────▶┐ ╰─────────────────────╯ │ ▼ ╭──────────────────────────────╮ │ N × TransformerBlock │ │ ┌─────────────────────────┐ │ │ │ x + MHA(LayerNorm(x)) │ │ ← residual + pre-norm │ │ x + FFN(LayerNorm(x)) │ │ │ └─────────────────────────┘ │ ╰──────────────────────────────╯ │ ▼ ╭──────────────────────────────╮ │ Final LayerNorm + lm_head │ ╰──────────────────────────────╯ │ ▼ logits (B, T, vocab_size) ``` Every layer (`Embedding`, `PositionalEmbedding`, `LayerNorm`, `Linear`, `ReLU`, `Softmax`, `AttentionHead`, `MultiHeadAttention`, `FeedForwardNetwork`, `TransformerBlock`, `GPT`) implements: - `forward(...)` — caches activations needed for backward - `backward(dout)` — manual analytic gradient, in-place - `parameters()` — returns list of `(param, grad)` references for the optimizer A causal (lower-triangular) mask is applied inside attention; cross-entropy and softmax are fused for a clean `dlogits = (softmax − one_hot) / N` gradient. ## Training stack | Component | What …