majoradmiral/nguzo.ai

Research programme on linguistically-informed tokenizers for low-resource African languages. Scope 1: Swahili BPE with 93% atomic-chunk coverage vs 50% vanilla. # Nguzo.ai > A research programme investigating whether low-resource African languages > are served better by language models that incorporate explicit linguistic > priors, rather than by scale alone. This repository is a unified > combination of the web application, the ML pipeline, and the supporting > research artifacts. --- ## What's in this repo | Source | What it is | Where it lives | |---|---|---| | `nguzo.ai--Language-Barrier-Bridge-` | AI Studio TypeScript/React web app | root (`src/`, `server.ts`, `vite.config.ts`, ...) | | `nguzo.ai` (Python side) | ML pipeline + Hadithi web app | `ml/`, `notebooks/`, `hadithi-app/`, `docs/` | | **Research artifacts (new)** | Scope-1 controlled tokenizer experiment, tests, results | `ml/`, `tests/`, `results/`, `RESEARCH.md` | **The research artefact is the centerpiece** - see **`RESEARCH.md`** for the research question, method, results, threats to validity, and the Scope-2 plan. Headline result: a linguistically-informed BPE tokenizer trained on Swahili-specific atomic chunks achieves **93% vocabulary coverage of the targeted chunks** versus **50% for a vanilla BPE** with the same data and vocabulary size. --- ## Repository layout ``` nguzo.ai/ ├── src/ # Root web app (TypeScript/React) ├── assets/ # Static assets ├── index.html # Root web app entry ├── package.json # Root web app deps ├── server.ts # Express dev server ├── tsconfig.json # TypeScript config (excludes hadithi-app, ml/) ├── vite.config.ts ├── metadata.json ├── .env.example │ ├── hadithi-app/ # Hadithi Vite/React app │ ├── src/{components,pages}/ │ ├── index.html │ ├── tailwind.config.js │ └── vite.config.ts │ ├── ml/ # Python ML modules │ ├── data_acquisition.py # Fetch small Swahili corpus │ ├── data-integration.py # Multi-source East-African data registry │ ├── east-african-tokenizer-config.py # Linguistic confi …