Duypq1612/CE-GCN

A Clone-and-Evolve Multi-Stream GCN framework for low-resource Vietnamese Sign Language Recognition, achieving 77.24% Top-1 accuracy and 245 FPS. # VSL-GCN **Official implementation of: Enhancing Vietnamese Sign Language Recognition via Cross-Modal Transfer Learning and Multi-Stream Ensemble based on CTR-GCN.** ## Introduction Vietnamese Sign Language (VSL) recognition faces significant hurdles due to the lack of large-scale labeled datasets and high inter-class similarity. Traditional methods often fail to capture the subtle nuances of VSL gestures. This project introduces **VSL-GCN**, a comprehensive framework that integrates a robust preprocessing pipeline with a novel training strategy. Our core contributions include: **1. Greedy Shape-Matching Transfer:** A novel weight loading mechanism that bypasses variable naming mismatches and enables partial channel loading, allowing the model to leverage >90% of pre-trained knowledge even when input channels differ (9-channel fusion vs. 3-channel original). **2. "Clone & Evolve" Strategy (Cross-Modal Initialization):** Instead of training Bone and Velocity streams from scratch or incompatible weights, we propose a cross-modal transfer method. We "clone" the weights from the converged Joint stream to initialize Bone and Velocity streams. This addresses the "cold-start" problem and significantly boosts the accuracy of secondary streams (e.g., Bone stream improvement from 41% to ~76%). **3. Robust Multi-Stream Ensemble with TTA:** A fusion framework that combines Joint, Bone, and Velocity streams using optimized weights, further enhanced by Test-Time Augmentation (TTA) to achieve state-of-the-art performance on our custom VSL dataset. ## Performance Experiments were conducted on our self-collected VSL Dataset containing 200 classes. | Method | Stream | Strategy | Top-1 Acc (%) | Top-5 Acc (%) | Checkpoint | | :--- | :---: | :--- | :---: | :---: | :---: | | Baseline (NTU Weights) | Joint | Direct Transfer | 73.36 | 85.12 | link | | Baseline (NTU Weights) | Bone | Direct Transfer | 41.67 | 62.40 | - | | **Proposed (Ours)** | **Joint** | **Shape-Matching** …