saran-io/tamil-tiny-gpt
Domain:
natural language processingRecord type:
model
Creator:
sar
Host:
A decoder-only Transformer trained from scratch on Tamil — what actually breaks in a low-resource NLP pipeline.
# Tamil Tiny GPT
A **decoder-only Transformer** (GPT-style) trained **from scratch** on Tamil text with PyTorch.
You own the full stack: corpus → BPE tokenizer → binary datasets → training → local inference.
**Repository:** github.com
**Blog-style write-up** (includes the same GitHub link for readers): `docs/blog-building-tamil-tiny-gpt.md`
---
## Overview
| Item | Detail |
|------|--------|
| Model | Causal LM, next-token prediction |
| Params | ~10.8M (6 layers, 6 heads, 384-d embeddings) |
| Context | Up to 128 tokens (`block_size`; auto-shrunk if data is tiny) |
| Tokenizer | BPE (`tokenizers`), trained on your corpus |
| Train | AdamW, warmup + cosine decay, train/val loss logging |
| Inference | `sample.py`: temperature + top-k sampling |
---
## End-to-end pipeline
```mermaid
flowchart LR
subgraph Data
A[data/raw/*.txt] --> B[clean_corpus.py]
B --> C[tamil_corpus.txt]
end
subgraph Tokenize
C --> D[train_tokenizer.py]
D --> E[tamil_bpe.json]
end
subgraph Prepare
C --> F[prepare_data.py]
E --> F
F --> G[train.bin / val.bin]
end
subgraph Train
E --> H[train.py]
G --> H
H --> I[ckpt.pt]
end
subgraph Generate
E --> J[sample.py]
I --> J
J --> K[Tamil text]
end
```
### Training loop (inside `train.py`)
```mermaid
flowchart TD
A[Load train.bin / val.bin memmap] --> B[Sample random windows]
B --> C[GPT: token + position embed]
C --> D[6 × Transformer block]
D --> E[LayerNorm + lm_head]
E --> F[Cross-entropy vs next token]
F --> G[Backward + AdamW + grad clip]
G --> H{Every N steps?}
H -->|yes| I[Eval train & val loss]
I --> J[Save checkpoint]
H -->|no| B
J --> B
```
### Model block (simplified)
```mermaid
flowchart TB
subgraph Block["Transformer block × n_layer"]
LN1[LayerNorm] --> ATT[Causal self-attention]
ATT --> ADD1[+ residual]
LN2[LayerNorm] --> MLP[MLP GELU]
MLP --> ADD2[+ residual]
end
IN[Input embeddings] --> Block
Block --> OUT[Final LayerNorm → logits]
```
---
## Quick start
```bash
git clone github.com …