qeinstein/adtc-llm-limited-hardware

A high level implementation of a llm quantitization algo, for the africa deep tech challenge, begun — 18|06|26 # Jamii Afya — Offline Clinical Advisor (ADTC 2026) **Domain:** Healthcare & Medical · **Languages:** English + Kiswahili · **Runtime:** llama.cpp / GGUF, CPU-only, 100% offline *Jamii Afya* ("community health") is an offline clinical **decision-support** assistant for community health workers and nurses in rural African clinics. It runs on a commodity 8 GB-RAM laptop with no GPU and no internet, answers in the language of the question, grounds answers in a curated WHO/IMCI knowledge base, and always surfaces **danger signs and when to refer**. It is decision support — not a diagnosis, and not a replacement for a clinician. > Built for the **Africa Deep Tech Challenge 2026 — The Laptop LLM Challenge.** --- ## Why this design wins the rubric The score is `0.50·S_acc + 0.30·S_perf + 0.20·S_eff − P_thermal`. The official `adtc-profiler` measures perf/memory by running **`llama-bench` on the raw GGUF** and accuracy via **lm-eval** — it never runs our app — and the **audit build has all SIMD disabled**. So: - **Small beats big here.** A 14B model (the plan we inherited) scores ~12/100 on efficiency and near-zero on throughput on a scalar build — it throws away half the score. We ship a fine-tuned **Qwen3-0.6B-Base** (Apache-2.0, Kiswahili-capable) at **Q4_0, 364 MB on disk → 527 MB peak RAM**, measured by the official profiler on a scalar x86 build: **20.33 tok/s (S_perf 100/100), S_eff 92.65/100, no thermal throttling**. - **Accuracy is recovered, not sacrificed:** WHO/IMCI **RAG grounding** + few-shot + a domain **LoRA** (Kiswahili + answer format), while the base model's general reasoning is preserved for lm-eval. - **Our numbers survive the audit.** We benchmark against a **scalar (no-SIMD) llama.cpp build** that mirrors the grading VM, so Gate-1 self-reports match the Gate-2 audit within tolerance — a variance-fail trap most teams miss. Full reasoning and the model A/B (0.6B vs 4B) are in **REPORT.md**. --- ## Architecture ``` Query (EN/SW) │ ▼ BM25 retri …