akolpaul2244/Sauti-Agent

Voice-driven telco/fintech agent for Swahili & Luganda-English code-switched speech — built for the MLC (Africa) × Intron Agentic Voice AI Challenge, benchmarked against Whisper & Gemini via Sahara. # Sauti Agent — MLC (Africa) × Intron Agentic Voice AI Challenge A voice-driven telco/fintech support agent that understands code-switched speech (Swahili/Luganda/English) and takes real actions — buying a data bundle or airtime, and reporting/reversing a failed mobile money transaction — instead of just transcribing. Category: **Fintech / Telco Call Center** Submitted by: Akol Paul (individual) ## How it works ``` [Mic recording in browser] ↓ (webm upload) [Django API] → Sahara (sync, fallback to async) / Whisper / Gemini ↓ [NLU: intent + entity extraction] ← phone number, amount, bundle size, txn id ↓ [Confirm-before-execute] ← customer confirms extracted details before anything runs ↓ [Mock telco action] → confirmation reply (spoken/shown) ``` The confirm step exists because our own benchmarking found that entity extraction (phone numbers especially) can vary between calls even on identical audio — see `Sauti_Agent_Benchmark_Report.pdf` for the full finding. A misheard digit shouldn't be able to fire a real transaction silently. The same transcript pipeline also powers `benchmark/run_benchmark.py`, which runs identical audio through all three models and reports word error rate (WER) plus — more importantly for fintech — whether each model got the **phone number** and **amount** right. ## Critical integration note: Sahara requires an explicit language code `use_language_asr_input` defaults to `"en"` server-side if omitted. Without it, Sahara returns a successful `FILE_TRANSCRIBED` status with an **empty** transcript — it fails silently, not loudly. Always pass the real language code (`sw` for Swahili, `lg` for Luganda). See `agent/services/intron.py` for the full confirmed endpoint behavior and language code table. ## Project layout ``` backend/ Django API agent/ services/ intron.py Sahara sync + async client (confirmed endpoints) other_stt.py Whisper (local) + Gemini nlu.py …