MohamedLouayChatti/TunSL_detection

Computer vision project for Tunisian Sign Language recognition # Tunisian Sign Language (TSL) Detection A machine learning project for recognizing Tunisian Sign Language using computer vision and deep learning techniques. This project implements a hybrid approach combining ResNet-based image classification with MediaPipe hand landmark detection and XGBoost classification to address domain generalization challenges. ## Project Overview This project tackles the recognition of Tunisian Sign Language (TunSL) using two different approaches: 1. **ResNet50-based Image Classification**: A traditional deep learning approach using transfer learning 2. **Hybrid Hand Landmark Detection + XGBoost**: A novel approach using MediaPipe for hand landmark extraction and XGBoost for classification The hybrid approach was developed to address the domain generalization problem encountered with the ResNet model, which tends to overfit to environmental factors (backgrounds, clothing, lighting) rather than focusing on actual hand gestures. ## Dataset This project uses the **(First ever) Tunisian Sign Language Dataset** by Chakroun, Wassim; Jerbi, Rihab (2023), available under Attribution 4.0 International (CC BY 4.0). ## Methodology ### Approach 1: ResNet50 Transfer Learning The initial approach used a pre-trained ResNet50 model with custom classification layers: **Architecture:** - Pre-trained ResNet50 backbone (frozen initially) - Custom classification head with dropout layers - Batch normalization for improved training stability - Data augmentation (rotation, translation, contrast, brightness, hue) **Results:** - High validation accuracy on same-distribution data - Potential overfitting to environmental factors rather than hand gestures ### Approach 2: Hybrid Hand Landmark Detection + XGBoost To address the domain generalization problem, we developed a hybrid approach: **Pipeline:** 1. **Hand Landmark Extraction**: MediaPipe Hand Landmarker detects 21 key points per hand 2. **Feature Engineering**: Extract world coordinates (x, y, z) …