prosperpiusmbaruku007-ship-it/AFRICA-GIANTS

Autonomous pipeline for building a Tanzanian/Swahili AI Assistant # AFRICA GIANTS A continuous LLM fine-tuning and redeployment pipeline for **Tanzanian Business Insights**. Combines QLoRA training on Kaggle GPUs with RAG at inference time to produce a bilingual (Swahili/English) AI assistant grounded in Tanzanian business knowledge. **Base model:** `McGill-NLP/AfriqueLlama-8B` — Llama 3.1 8B pre-trained on 20 African languages including Swahili ## Architecture ``` Web Scraping (TRA, BRELA, NBS, BoT, ...) ↓ Data Gate → Clean → Deduplicate → Vector DB (RAG index) ↓ Synthetic Q&A Generation ↓ SFT JSONL → Hugging Face Dataset Upload ↓ Kaggle QLoRA Training → Hugging Face Model Upload ↓ Evaluation Gate → Model Registry → FastAPI Serving ↓ Monitoring + Feedback → next training cycle ``` **Two-layer inference:** - **Fine-tuning** — teaches behavior, tone, domain reasoning - **RAG** — provides fresh factual knowledge at inference time ## Dataset — adapter-v5 | | Count | |---|---| | **Total cleaned pairs** | 2,152 | | **Training pairs** (SFT) | 1,927 | | **Validation pairs** | 215 | | **Eval-set pairs** (held out) | 10 | | **Cleaned batches** | 12 (batch_001 – batch_010) | | **HuggingFace dataset** | prospAprospA007/africa-giants-dataset | **batch_010 (90 pairs, adapter-v5 targeted fixes):** - 40 refusal-training pairs covering 8 out-of-corpus domains that had zero coverage in adapter-v4 - 5 PAYE band-by-band calculation pairs (correct figures) - 25 GN 605A disambiguation pairs (33.4% vs 33.5%, average vs maximum, date disambiguation, 2022 order revocation) - 20 SDL quick-win pairs (deadline, cash-only base, threshold mechanics, SDL vs WCF distinction) ## Quickstart ```bash # Install dependencies pip install -r requirements.txt # Copy and fill in credentials cp .env.example .env # Collect and prepare training data python run.py scrape # Start inference server python run.py serve --port 8000 ``` ## Commands | Command | Description | |---|---| | `python run.py scrape` | Scrape → clean → deduplicate → generate synthetic Q&A | …