wongeld/agri-voice-realtime-rw

Real-time Kinyarwanda agri voice assistant backend: live ASR (Whisper), multi-intent parsing with conditional handling, and MMS TTS, plus a hands-free web UI and initial experiment notebook. # Voice Backend (Live ASR -> Intent -> TTS) This repository contains two parts: 1. Initial notebook experiments for ASR and TTS in `asr.ipynb` 2. Production-style FastAPI backend in `backend/` This backend is focused only on: 1. Live audio ingestion 2. Kinyarwanda transcription with `faster-whisper` 3. Agriculture intent parsing (supports conditional clauses and multiple intents) 4. Text generation per detected intent and Kinyarwanda TTS synthesis Optional agentic mode is now supported using Groq + LangChain. If enabled, responses are generated from live sensor readings fetched from configured GET endpoints. Sensor collection, dashboards, and hardware calls remain out of scope and should be handled by your other API. ## Initial Notebook Experiments The notebook `asr.ipynb` captures the early validation flow before backend refactoring: 1. ASR prototype with `faster-whisper`: - Installs and loads `WhisperModel("small", device="cpu", compute_type="int8")` - Transcribes `audio_test_1.ogg` - Notes observed speed from initial run 2. TTS prototype with Meta MMS model: - Loads `facebook/mms-tts-kin` (`VitsModel` + `AutoTokenizer`) - Generates waveform from Kinyarwanda text - Saves output audio to `output.wav` 3. Early API sketch: - Includes an initial `/voice` endpoint concept in notebook cells - This was later replaced with the modular FastAPI implementation under `backend/` If you want to re-run the notebook experiments directly, open `asr.ipynb` and execute the code cells in order. ## Repository Layout - `asr.ipynb`: initial ASR and TTS experimentation notebook - `audio_test_1.ogg`: sample input used during early ASR tests - `output.wav`: generated sample output from notebook TTS test - `backend/`: current real-time ASR -> intent -> TTS service and web UI ## Endpoints - `GET /health` - `GET /` web UI for live demo - `POST /voice` upload one audio file (expects 16kHz audio) - `WS /ws/live` send PCM16 mono 16kHz chunks and receive transcript+intent+tts turn …