SegniNegasa123/Tereguwami---

n Adaptive Multimodal AI System for Two-Way Communication Between Ethiopian Sign Language Users and Non-Signers # Tereguwami — ተርጓሚ ### An Adaptive Multimodal AI System for Two-Way Communication Between Ethiopian Sign Language Users and Non-Signers --- ## 1. Overview **Tereguwami** (ተርጓሚ, Amharic for *"the translator/interpreter"*) is the first open, benchmarked, bidirectional communication system designed for **Ethiopian Sign Language (ESL / ETHSL)**. It bridges communication barriers between Deaf/hard-of-hearing individuals and hearing non-signers across vital public and private domains: healthcare, education, legal proceedings, banking, public broadcasts, and everyday life. Tereguwami unifies four core research threads into a single deployed accessibility platform: 1. **Continuous, Non-Manual-Marker-Aware Translation**: Sentence-level recognition mapping hands, body pose, and facial grammatical markers (eyebrow, mouth, head movements) into fluent Amharic, Afaan Oromo, and English. 2. **Generative Sign Production (Reverse Channel)**: Continuous avatar performance generated directly from text/speech using sequence transformers. 3. **Non-Invasive Neuromuscular "Silent Speech" Channel**: AlterEgo-derived sEMG decoding from facial/jaw subvocalizations for camera-free communication in low-light, occupied-hand, or privacy-sensitive contexts. 4. **Companion Wearable Control Interface**: Wrist-worn surface-EMG band for hands-free, silent command and control. --- ## 2. Nine-Layer System Architecture ```mermaid graph TD subgraph Input ["Perception & Input Layers"] V[Video Feed] --> L1[8.1 Perception Layer MediaPipe Holistic / Keypoints] EMG1[Jaw/Face sEMG] --> L7[8.7 Silent-Speech Layer Subvocalization Decoder] EMG2[Wrist sEMG] --> L8[8.8 Companion Wearable Silent Control Band] end subgraph Core ["Recognition & Translation Backbone"] L1 --> L2[8.2 Recognition Layer Temporal Sequence Model] L1 --> L3[8.3 Non-Manual Marker Layer Facial Grammar & Semantics] L2 --> L4[8.4 Translation / Language Layer Gloss-Free Transformer + Multilingual Decoder] L3 --> L4 L7 --> L4 end subg …