soufianeboulahcen/Master_Multimodal-Moroccan-SLG

Multimodal Moroccan Sign Language Generation MoSL # Multimodal Moroccan Sign Language Generation A paper-faithful PyTorch reimplementation of **SignLLM** trained on the **MoSL** isolated-word dataset, extended with a full OpenPose-style skeleton tracking and video generation pipeline. --- ## Overview | Phase | Description | |-------|-------------| | Dataset processing | Label extraction, train/val/test split, vocabulary | | Pose extraction | OpenPose keypoint extraction to NPZ per clip | | 2D to 3D lifting | Prompt2Sign `.skels` format conversion | | Model | SignLLM encoder-decoder transformer (35M params) | | Training | MSE / RL / RL+PLC ablation runs | | Evaluation | Teacher-forced MSE + autoregressive DTW vs baselines | | Visualization | OpenPose-style skeleton videos, GIFs, JSON keypoints | --- ## Results **Test AR DTW (lower is better):** | Method | DTW | |--------|-----| | Nearest-Neighbor (baseline) | 0.7817 | | Mean-Pose (baseline) | 0.8682 | | Random-Clip (baseline) | 0.9620 | | SignLLM MSE | 1.0447 | | SignLLM RL+PLC | 1.2117 | | SignLLM RL | 1.3182 | --- ## Generated Outputs ### Skeleton Tracking Videos Six motion types with full body, hand, and face tracking (1280x720, 30 fps): | Motion | Skeleton | Overlay | Slow-motion | |--------|----------|---------|-------------| | Walking | `outputs/videos/skeleton/walking_skeleton.mp4` | `outputs/videos/overlay/walking_overlay.mp4` | `outputs/videos/slowmo/walking_slowmo.mp4` | | Running | `outputs/videos/skeleton/running_skeleton.mp4` | `outputs/videos/overlay/running_overlay.mp4` | `outputs/videos/slowmo/running_slowmo.mp4` | | Dancing | `outputs/videos/skeleton/dancing_skeleton.mp4` | `outputs/videos/overlay/dancing_overlay.mp4` | `outputs/videos/slowmo/dancing_slowmo.mp4` | | Jumping | `outputs/videos/skeleton/jumping_skeleton.mp4` | `outputs/videos/overlay/jumping_overlay.mp4` | `outputs/videos/slowmo/jumping_slowmo.mp4` | | Hand & Face | `outputs/videos/skeleton/hand_face_skeleton.mp4` | `outputs/videos/overlay/hand_face_overlay.mp4` | `outp …