henok555/ethiopic-tokenization

# Consonantal Factorization Before Byte-Pair Encoding Improves Token Efficiency and Root Consistency in Amharic Tokenization Standard BPE tokenizes Ethiopic (Amharic) text purely by surface frequency, with no awareness that Ethiopic's syllabic script encodes a separable consonant/root and vowel/order structure. This project builds a tokenizer that factorizes Ethiopic characters into those two streams, runs BPE on the consonant/root stream, and selectively merges that representation back with standard BPE (a "hybrid" tokenizer) word by word. Across a ~8.5M-character Amharic corpus, the hybrid tokenizer uses **~30% fewer tokens per word** than standard BPE while exhibiting higher root-token consistency. A small transformer language model trained on this tokenization achieves a **17–21% reduction in bits-per-character**, an advantage that persists under character- and FLOP-matched training budget regimes. ## Author Henok Woldebirhan ## Repository Structure ``` config/ Experiment configuration (merge-count sweeps, LM hyperparameters, seeds) data/ Training, validation, and testing corpus files data_pipeline/ Data cleaning and pre-processing utilities src/ Core library: tokenizer, BPE, hybrid selection, LM, metrics scripts/ Pipeline stage drivers (00_data_audit.py through 09_ablation_extended.py) notebooks/ Colab runner notebook for end-to-end execution paper/ LaTeX manuscript source and figures results/ Evaluation metrics, plots, and comprehensive RESULTS_SUMMARY.md tests/ Unit test suite covering losslessness, edge cases, and selection logic logs/ Trained-tokenizer disk cache and run configs ``` ## Setup & Quickstart ```bash pip install -r requirements.txt ``` Place `amharic.train.clean.text`, `amharic.dev.clean.text`, and `amharic.test.clean.text` in `data/`. ### Running Unit Tests ```bash python -m pytest tests/ -q ``` ### End-to-End Reproduction Workflow Execute the pipeline scr …