African-Languages-Lab/mansa-nano-inference
Domaine:
natural language processingType de record:
softwaremodel
Créateur:
Afr
Hôte:
Minimal inference-only package for mansa-nano: pulls weights from Hugging Face and serves a web UI + chat API.
# mansa-nano — inference
A minimal, inference-only package for **mansa-nano**, a small English-centric chat model
trained by Sheriff Issaka and the African Languages Lab as part of a ~$100 training
experiment. This repo contains only what is needed to *serve* the model — no training,
data, or evaluation code. Weights are pulled from the Hugging Face Hub on startup.
## Quickstart
```bash
# 1. (recommended) create a fresh environment
python -m venv .venv && source .venv/bin/activate
# 2. install deps (see requirements.txt for a hardware-specific torch build)
pip install -r requirements.txt
# 3. serve the web UI + API (auto-downloads weights from Hugging Face)
python -m scripts.serve
```
Then open localhost in your browser.
Runs on CPU, a single GPU (CUDA), or Apple Silicon (MPS) — the device is autodetected.
bf16 weights are automatically cast to fp32 on CPU/MPS.
## Command-line chat
```bash
python -m scripts.chat_cli -p "What is the capital of France?"
```
## API
`POST /chat/completions` (streaming, Server-Sent Events):
```bash
curl -N localhost \
-H 'Content-Type: application/json' \
-d '{"messages":[{"role":"user","content":"What is the capital of France?"}]}'
```
Other endpoints: `GET /` (web UI), `GET /logo.svg`, `GET /health`, `GET /stats`.
## Configuration
Common flags (both `scripts/serve.py` and `scripts/chat_cli.py`):
| Flag | Default | Description |
| --- | --- | --- |
| `--hf-repo` | `African-Languages-Lab/mansa-nano` | HF repo hosting the weights |
| `--local-dir` | _(none)_ | Load from a local checkpoint dir, skip HF download |
| `--step` | latest | Checkpoint step to load |
| `-t/--temperature` | `0.8` (serve) / `0.6` (cli) | Sampling temperature |
| `-k/--top-k` | `50` | Top-k sampling |
| `-m/--max-tokens` | `512` | Max tokens per response |
| `--system-prompt` | persona | Persona prepended to first user turn (`""` to disable) |
| `--device-type` | autodetect | `cuda` \| `cpu` \| `mps` |
`scrip …