saran-io/tamil-tiny-gpt

A decoder-only Transformer trained from scratch on Tamil — what actually breaks in a low-resource NLP pipeline. # Tamil Tiny GPT A **decoder-only Transformer** (GPT-style) trained **from scratch** on Tamil text with PyTorch. You own the full stack: corpus → BPE tokenizer → binary datasets → training → local inference. **Repository:** github.com **Blog-style write-up** (includes the same GitHub link for readers): `docs/blog-building-tamil-tiny-gpt.md` --- ## Overview | Item | Detail | |------|--------| | Model | Causal LM, next-token prediction | | Params | ~10.8M (6 layers, 6 heads, 384-d embeddings) | | Context | Up to 128 tokens (`block_size`; auto-shrunk if data is tiny) | | Tokenizer | BPE (`tokenizers`), trained on your corpus | | Train | AdamW, warmup + cosine decay, train/val loss logging | | Inference | `sample.py`: temperature + top-k sampling | --- ## End-to-end pipeline ```mermaid flowchart LR subgraph Data A[data/raw/*.txt] --> B[clean_corpus.py] B --> C[tamil_corpus.txt] end subgraph Tokenize C --> D[train_tokenizer.py] D --> E[tamil_bpe.json] end subgraph Prepare C --> F[prepare_data.py] E --> F F --> G[train.bin / val.bin] end subgraph Train E --> H[train.py] G --> H H --> I[ckpt.pt] end subgraph Generate E --> J[sample.py] I --> J J --> K[Tamil text] end ``` ### Training loop (inside `train.py`) ```mermaid flowchart TD A[Load train.bin / val.bin memmap] --> B[Sample random windows] B --> C[GPT: token + position embed] C --> D[6 × Transformer block] D --> E[LayerNorm + lm_head] E --> F[Cross-entropy vs next token] F --> G[Backward + AdamW + grad clip] G --> H{Every N steps?} H -->|yes| I[Eval train & val loss] I --> J[Save checkpoint] H -->|no| B J --> B ``` ### Model block (simplified) ```mermaid flowchart TB subgraph Block["Transformer block × n_layer"] LN1[LayerNorm] --> ATT[Causal self-attention] ATT --> ADD1[+ residual] LN2[LayerNorm] --> MLP[MLP GELU] MLP --> ADD2[+ residual] end IN[Input embeddings] --> Block Block --> OUT[Final LayerNorm → logits] ``` --- ## Quick start ```bash git clone github.com …