Madeeha923/SignMotion

Signvrse is an end-to-end Hybrid AI system that translates natural English text into real-time 3D Kenyan Sign Language (KSL) animations. It uses an LLM for translation and a custom PyTorch architecture (Autoencoder + Bi-LSTM) to generate mathematically accurate skeletal movements directly in the browser. # Signvrse: English to 3D Kenyan Sign Language Translator **Live Web Demo:** Signvrse on Vercel **Cloud Backend:** Hugging Face Space --- ## Overview **Signvrse** is an end-to-end Hybrid AI engine that translates natural English into real-time 3D Kenyan Sign Language (KSL) animations. It utilizes a LangGraph agentic workflow to convert English to KSL gloss and intelligently routes the request to either a FAISS-based retrieval database or a custom PyTorch generative pipeline. This custom deep learning architecture, trained on over 12,000 spatial-temporal sequences, uses a decoupled Autoencoder and Bi-LSTM to dynamically synthesize exact 3D skeletal coordinates for unseen phrases. The system completely bypasses heavy video rendering by streaming these raw mathematical joint rotations directly to a React Three Fiber frontend, instantly animating a 3D avatar at a seamless 60 frames per second. ## Key Features * **Real-Time 3D Rendering:** Bypasses heavy MP4 generation by rendering motion mathematically in the browser at 60 FPS. * **Agentic NLP Translation:** Uses an LLM agent to accurately translate English grammar into proper KSL Gloss (dropping articles, restructuring verbs). * **Hybrid RAG Pipeline:** Intelligently searches a vector database for authentic human motion sequences before attempting to generate new ones. * **Dynamic Generative Fallback:** Uses a custom-trained Autoencoder and Bi-LSTM to invent mathematically accurate 3D skeletal movements for completely unseen vocabulary. * **Device Agnostic:** Runs entirely in the cloud, meaning the 3D web interface works instantly on mobile phones, tablets, and low-end laptops. ## Tech Stack * **Frontend:** React.js, Vite, React Three Fiber, Drei (WebGL). * **Backend:** Python, FastAPI, LangGraph, OpenAI API. * **Machine Learning:** PyTorch, FAISS, NumPy, Pandas. * **Deployment:** Vercel (Frontend), Hugging Face Spaces & Docker (Backend). --- ## How It Works (The Pipeline) ### 1. The Data & Training The fou …