Rashiidmataan/somali-annotation-dashboard

# Somali Native Speech Dataset Pipeline A local dashboard and processing pipeline for collecting Somali native audio, splitting long recordings into short clips (2–30s), annotating transcripts, running quality checks, and exporting a training-ready dataset for ASR and speech models. **Goal:** Build a fast, consistent, and scalable system to reach large volumes (e.g., 5,000+ hours) without manual chaos. --- ## 1. What this system does ### Inputs - Long Somali audio recordings (minutes to hours) - Source info (where the audio came from, date, type) ### Outputs - Short clips: **2s to 30s** - One transcript per clip (Somali text) - Metadata manifest for training (CSV/JSON/Kaldi-style) Each clip is saved with a unique ID, and the transcript is saved using the **same ID**. Example: - `clips/SOM_000123.wav` - `texts/SOM_000123.txt` --- ## 2. Core workflow (end-to-end) ### Step A: Intake (collect & register) 1. Download / gather long audios. 2. Register each audio file in the system with basic metadata: - source name (radio/podcast/interview/etc.) - date - domain/topic (news, conversation, lecture, poetry, etc.) - consent/license status (if applicable) **Output:** raw audio is stored in `data/raw/` and referenced in metadata. --- ### Step B: Normalize (standardize audio) All audio is converted to a consistent training format: - WAV - mono - 16kHz sample rate **Output:** normalized copies stored in `data/normalized/` --- ### Step C: Auto-segmentation (silence-based split) Long audio is split into candidate segments using **silence detection**: - Detect silent gaps and cut around them - Enforce: - `min_length = 2s` - `max_length = 30s` Rules: - Segments shorter than 2s are merged with neighbors when possible - Segments longer than 30s are split further into smaller chunks (e.g., 15–20s windows) **Output:** segment candidates stored in `data/segments/` + segment list in the database/manifest. --- ### Step D: Annotation (human-in-the-loop) Annotators use th …