sonyarko9/-gsl-stgcn-pipeline

Real-time Speech-to-Ghana Sign Language generation pipeline for CS/IT lecture delivery — KNUST 2025/2026 # -gsl-stgcn-pipeline Real-time Speech-to-Ghana Sign Language (GSL) generation pipeline for CS/IT lecture delivery — KNUST 2025/2026 final-year capstone. Supervised by Dr. Osei. Proposal scored 131/145 (Distinction). --- ## Overview This pipeline converts spoken CS/IT lecture audio into real-time GSL sign output, via four stages: 1. **Whisper ASR** (`transcriber.py`) — speech → text. ✅ Done, working. 2. **Hybrid CGOP gloss mapper** (`gloss_mapper.py`) — text → gloss tokens, using a deterministic dictionary lookup + consistency cache, falling back to Mistral-7B via Ollama (CPU-only, chosen due to GTX 1650 4GB VRAM constraint) for terms not in the dictionary. ✅ Done, working. See Gloss Dictionary below. 3. **ST-GCN + multi-head attention classifier** (4-head attention, `d_model=256`, 75-node MediaPipe adjacency matrix) — gloss → sign sequence. 🚧 Architecture designed, **not yet trained**. 4. **OpenCV stick-figure avatar renderer** — sign sequence → rendered animation. ✅ Done, working. --- ## Dataset Status - **Target (long-term):** 79 signs × 3 signers × 3+ reps, recorded on green or yellow screen, MediaPipe Holistic keypoints, shape `(T, 225)`. - **Ethics:** CHRPE clearance required before any real signer recording. Institutional partners: GNAD, ARAD. - **Current dataset:** 1 signer, 3 reps per gloss, **79 signs** recorded as raw video (not the original 90 — scope was narrowed). This is being treated as a **pipeline smoke test**, not final defense-grade training data. - **Near-term target (pending recruitment):** 3 signers × 80 signs × 10 reps (2,400 clips), to enable 3-fold Leave-One-Signer-Out (LOSO) cross-validation. - **Important:** synthetic/augmented data must **never replace** real data for the defense. Augmentation (temporal warp, frame dropout, jitter, rotation, scaling — see Colab notebook) supplements real data only. --- ## Naming Convention - **Video files:** `GLOSS_signerXX_repXX.mp4` (e.g. `HARD-DRIVE_signer01_rep01.mp4`) - **Folder per glos …