abubakar-xyz/Sabi-AI

Sabi AI - Your Naija AI Bestie # 🌟 SABI — Your Naija AI Bestie SABI is a real-time, voice-to-voice multimodal AI companion. Built with bidirectional audio streaming, SABI pairs conversational intelligence with a dynamic 3D avatar that visually reacts to both your voice input and the AI’s spoken responses. --- ## ⚡ Tech Stack * **Frontend:** React, Vite, Tailwind CSS (Glassmorphism UI) * **3D Canvas:** Three.js via `@react-three/fiber` & `@react-three/drei` * **Audio Layer:** Web Audio API (`AudioContext`, PCM streaming, Jitter Buffering) * **Backend:** Node.js, Express, `ws` (WebSockets) * **AI Engine:** Google Gemini (Multimodal Live API via `@google/genai`) --- ## 🏗️ System Architecture ``` [ Microphone ] ──(16kHz PCM)──> [ React Client ] │ (WebSocket / Base64) ▼ [ Node.js Relay ] │ (Gemini Live SDK) ▼ [ Google Gemini Live API ] │ (24kHz PCM Chunks) ▼ [ Node.js Relay ] │ (WebSocket / Base64) ▼ [ 3D Avatar (Lerp) ] [ Speaker Output ] ``` ### 1. The Client (React + Three.js) * **Audio Input:** Captures microphone input via `navigator.mediaDevices.getUserMedia()` and samples down to 16,000 Hz using `ScriptProcessorNode`. * **Audio Output:** Receives 24,000 Hz PCM chunks, decodes them via `base64ToPcm`, and queues them through an audio jitter buffer to prevent crackling. * **3D Engine:** Calculates live amplitude (`speakerLevel`) and applies linear interpolation (`THREE.MathUtils.lerp`) inside the `useFrame` loop to animate the avatar's scale, aura, and mouth movements. ### 2. The Relay Server (Node.js + WebSockets) * Keeps your `GEMINI_API_KEY` secure on the server side. * Establishes a bidirectional WebSocket proxy between the browser and Google's Live API stream. ### 3. The Brain (Gemini Multimodal Live API) * Maintains a persistent socket session. * Listens continuously to audio frames, performs real-time turn-taking detection, and streams synthesized voice chunks back with near-zero latency. --- ## 🚀 Getting Started ### Prerequisites * Node.js (v18+) or Bun * A Google Gemini …