i-am-christy/speech-to-sign-backend

A real-time AI application that translates spoken audio into Nigerian Sign Language (NSL) animations. It utilizes OpenAI Whisper for speech-to-text, a custom Progressive Transformer model for text-to-sign translation, and React Three Fiber for 3D avatar visualization. # SignBridge: Audio-to-Sign Language Translator SignBridge is a machine learning-powered web application designed to bridge the communication gap between the hearing and the deaf communities in Nigeria. It translates spoken audio into 3D Nigerian Sign Language (NSL) animations in real-time. The system works by first transcribing audio using OpenAI's Whisper model, translating the text into sign language pose data using a custom Progressive Transformer model, and finally visualizing the result on a 3D avatar built with Three.js. ## System Architecture The project consists of two main components: 1. **Backend (Python/FastAPI):** Handles audio processing, speech recognition, and deep learning inference to generate skeletal pose data. 2. **Frontend (React/Vite):** A modern user interface that captures audio input and renders the resulting 3D animation using WebGL. ## Key Features * **Speech-to-Text:** Robust audio transcription using OpenAI Whisper. * **Text-to-Sign Translation:** Neural Machine Translation (NMT) model trained specifically for Nigerian Sign Language structure. * **3D Visualization:** Real-time rendering of a skeletal rig using React Three Fiber. * **Audio Recording:** Native browser support for recording voice input directly. * **File Upload:** Support for processing pre-recorded `.wav` or `.mp3` files. ## Technology Stack * **Backend:** Python, FastAPI, PyTorch, TorchText, OpenAI Whisper, Uvicorn. * **Frontend:** React (Vite), React Three Fiber (R3F), Drei, Tailwind CSS, Lucide React, Axios. * **Machine Learning:** Transformer-based architecture (Progressive Transformers) for Sign Language Production (SLP). ## Installation and Setup ### Prerequisites * Python 3.10 or higher * Node.js (v16 or higher) * NVIDIA GPU (Recommended for faster inference, but CPU is supported) ### 1. Backend Setup Navigate to the backend directory and set up the Python environment. ```bash cd backend # Create a virtu …