sathya8998/Hierarchical-Text-to-Sign-Motion-Generation-with-RVQ-VAE-and-Transformer

Hierarchical Text-to-Sign Motion Generation with Masked Residual Transformers.Implements a two-stage Masked + Residual Transformer architecture to generate 6-layer RVQ motion tokens from text using CLIP embeddings, featuring EMA-stabilized training, classifier-free guidance, and Gumbel sampling for realistic Kenyan Sign Language 3D avatar animation # HIERARCHICAL TEXT-TO-SIGN MOTION GENERATION WITH RVQ-VAE AND TRANSFORMERS ## PROJECT OVERVIEW This repository implements a state-of-the-art generative pipeline for the **Motion-S: Text-to-Sign Motion Generation** competition. The model translates English sentences or glossified text into realistic 3D sign language animations by generating **hierarchical motion tokens** (6 RVQ layers) that can be decoded into fluid Kenyan Sign Language (KSL) avatar movements. - **Accessibility-Focused Generative AI**: Bridges the communication gap for over 430 million deaf individuals by producing culturally authentic, smooth sign language motions. - **Hierarchical Token Generation**: Uses a two-stage Masked Transformer + Residual Transformer architecture conditioned on CLIP text embeddings. - **EMA-Stabilized Training**: Applies Exponential Moving Average for stable and high-quality generation. - **Fixed VAE Backbone**: Strictly adheres to the provided `rvq_vae_best.pth` tokenizer (no modifications allowed) to ensure compatible token outputs. - **Batched Inference**: Optimized batch processing for efficient generation of 3,000 test samples within Kaggle runtime limits. ## GOAL Generate **6 layers of motion tokens** (`base_tokens` + `residual_1` to `residual_5`) for each test prompt. Each token ∈ **[0, 511]** and all layers must have identical sequence lengths (40–800 tokens). ## DATASET - **Training Data**: ~15,000 text-motion pairs from Kenyan Sign Language (KSL). - **Input**: Natural English sentences and/or glossified text. - **Output**: Pre-tokenized 6-layer RVQ tokens (using the fixed `rvq_vae_best.pth`). - **Competition Link**: kaggle.com ## MODEL ARCHITECTURE The system uses a powerful two-stage transformer-based generative model: - **Text Encoder**: CLIP ViT-B/32 for rich text embeddings. - **Base Layer**: Masked Transformer with self-attention + cross-attention (SwiGLU FFN + S …