ZiFutures/superbpe-african-tokenizer

# SuperBPE for African Language Models End-to-end pipeline for training and evaluating SuperBPE tokenizers for eight African languages: Swahili, Zulu, Yoruba, Igbo, Hausa, Amharic, Somali, and Wolof. Full specification: see `Technical_Implementation_Document_v1.1.docx`. ## Setup ```bash conda env create -f environment.yml conda activate superbpe ``` ## Pipeline stages | Step | Command | Depends on | |---|---|---| | 1 | `python scripts/01_download_data.py` | — | | 2 | `python scripts/02_preprocess.py` | Step 1 | | 3 | `python scripts/03_train_bpe.py` | Step 2 | | 4 | `python scripts/04_train_superbpe.py` | Step 3 | | 5 | `python scripts/05_evaluate.py --type [fertility\|morpheme\|downstream]` | Step 3+4 | | 6 | `python scripts/06_package_and_upload.py` | Step 4 | ## Team See "Team & Roles" in the technical implementation document.