VNW22/voiceassist_api

A real-time speech-to-text and text-to-speech API built for speech-impaired users in Kenya. Powered by a finetuned Whisper ASR model and custom TTS pipeline for Kenyan English and Swahili.” # 🎙️ Voiceassist-API This repository hosts a high-performance, real-time Speech-to-Text (STT) and Text-to-Speech (TTS) backend powered by **Modal**. It is designed to handle streaming audio input and provide instant transcription and voice responses in **Kenyan English** and **Swahili**. --- ## 🏗️ Architecture & Models ### Speech-to-Text (STT) The system uses fine-tuned variants of OpenAI's Whisper (Medium) architecture, specifically optimized for non-standard dialects: * **English Model:** `Veronica1NW/en_whisper_nonstandard_medium` * **Swahili Model:** `Veronica1NW/sw_whisper_nonstandard_medium` ### Text-to-Speech (TTS) Audio responses are generated using **Azure Cognitive Services Speech SDK**, utilizing high-quality Neural voices specific to the region. ### Infrastructure * **Host:** Modal * **Hardware:** NVIDIA L4 GPU * **Timeout:** Sessions have a **strict 30-minute limit**. After 30 minutes, the WebSocket will disconnect, and the client must initiate a new connection. --- ## 🗺️ System Architecture Diagram ```mermaid graph TD; A[Client Browser / App] -->|16kHz PCM Audio| B(WebSocket Endpoint); B --> C[Modal GPU Function - STT Processor]; C -->|Transcribed Text| D[TTS Generator - Azure Speech]; D -->|WAV Audio Bytes| A; C -->|Text Frames| A; ``` --- ## 🌍 Supported Languages & Voices The backend supports dynamic switching between languages and genders during the stream. | Language | Code | Gender | Azure Voice ID | | :------------------- | :--- | :----- | :--------------------- | | **English (Kenyan)** | `en` | Male | `en-KE-ChilembaNeural` | | **English (Kenyan)** | `en` | Female | `en-KE-AsiliaNeural` | | **Swahili (Kenyan)** | `sw` | Male | `sw-KE-RafikiNeural` | | **Swahili (Kenyan)** | `sw` | Female | `sw-KE-ZuriNeural` | --- ## 🔌 Connectivity ### Endpoints * **Web Endpoint:** `veronicahwags-cdli--streami…` * **WebSocket URL:** `wss://veronicahwags-cdli--stre …