Daemon22/xnlp

xNLP - Xhosa Natural Language Processing toolkit # XNLP Core LLM - isiXhosa Language Model ## Overview This project contains the **XNLP Core Large Language Model**, a custom LLaMA-style transformer specifically designed for isiXhosa language processing, along with a professional, independent training pipeline (`xnlp_trainer`). ### Key Design Principle: Single-File Checkpoints Every model artifact is a **single self-contained `.pt` file** — model weights, model configuration, tokenizer state, optimizer state, scheduler state, and training metadata are all bundled together. There are no scattered tokenizer `vocab.json`, `merges.json`, or `config.json` files to manage. Simply copy one `best_model.pt` and it works anywhere. ## Package Architecture ``` xnlp/ ├── README.md # This file ├── core_llm/ # Model architecture (inference only) │ ├── __init__.py # Public API: XNLPCoreLLM, XNLPConfig, XNLPTokenizer │ ├── architecture.py # LLaMA-style transformer (RoPE, GQA, SwiGLU, RMSNorm) │ └── tokenizer.py # BPE tokenizer with special tokens ├── xnlp_trainer/ # Professional training pipeline (independent) │ ├── __init__.py # Public API + version │ ├── __main__.py # Enables `python -m xnlp_trainer` │ ├── config.py # TrainingConfig, model presets, checkpoint validation │ ├── data.py # Corpus loading, dataset, tokenizer serialization │ ├── trainer.py # XNLPTrainer: core training engine │ ├── evaluate.py # Loss, perplexity, sample generation, eval suite │ ├── inference.py # XNLPPredictor: single-file model loading │ └── run.py # CLI entry point with argparse ├── corpus/ # isiXhosa source text (authoritative) │ ├── mqhayi_complete.txt # Mqhayi poems and literary works │ ├── mqhayi_basic.txt # (synthetic - excluded from training) │ └── masikhanyise_complete.txt # Masikhanyise textbook content ├── co …