fishmisgh-crypto/sign_language

ESLT: Ethiopian Sign Language Translator - mobile app, ML training pipeline, and shared vocabulary # HandSign — Hardware-Free Sign Language Translator **HandSign** is a camera-based, hardware-free sign language translation application. It uses smartphone video to recognize handshapes and hand movements with instant speech synthesis in English, Amharic fidel, and Afaan Oromo. **Current Status:** - ✅ **Android**: Full on-device pipeline. MediaPipe hand tracking runs in a local native module (`mobile/modules/hand-landmarks`) as a VisionCamera frame processor. - ✅ **Web**: Full pipeline, with MediaPipe Hands loaded from CDN. --- ## 1. Core Architecture & Pipeline ``` [ Smartphone Camera (Front / Rear) ] │ ▼ [ Real-Time MediaPipe Hand Landmarker (21 3D Points) ] │ ▼ [ Geometry Normalization & Chirality Invariance (63 Features) ] │ ┌────────┴───────────────────────────┐ ▼ ▼ [ Static Alphabet Classifier ] [ MotionTracker (movement burst → resample to T=32) ] (Dense 63 → 256 → 128 → 64 → 28) │ (Displacement Velocity ≥ 0.018) │ ▼ │ [ 1D Temporal CNN Classifier ] │ (Conv1D → Pool → Conv1D → GAP → Dense → 6) │ │ ▼ ▼ [ LetterGate (Static Smoothing) ] [ TemporalGate (proposes; never auto-commits) ] └────────┬───────────────────────────┘ ▼ [ SpellingBuffer (A–Z, DEL, SPACE + tap-accepted whole words) ] │ ▼ [ Trilingual Dictionary Matching (English ↔ Amharic ↔ Afaan Oromo) ] │ ▼ [ Text-to-Speech (TTS) + Local History Storage ] ``` --- ## 2. Machine Learning & Computer Vision Specifications ### 2.1 Landmark Contract & Preprocessing - **Raw Input**: 21 MediaPipe hand landmarks with 3D spatial coordinates $(x_i, y_i, z_i)$, totaling 63 floating-point values. - **Translation Invariance**: Landmark 0 (wrist) is translated to $(0, 0, 0)$. - **Scale Invariance**: Coordinates are normalized by the maximum Euclidean distance from the origin to any landmark: $$\text{scale} = \max …