mohamed-okasha/arabic-dailect-asr-pipeline

Builds a labelled audio dataset for training Arabic-dialect ASR (Tunisian, Libyan, Levantine .. etc) by mining short videos that have **burned-in Arabic captions**. The captions provide the transcript; ffmpeg VAD provides the speech segmentation; an Arabic-aware vision-language model OCRs the captions # Arabic-dialect ASR data pipeline Builds a labelled audio dataset for training Arabic-dialect ASR (Tunisian, Libyan, etc.) by mining short videos that have **burned-in Arabic captions**. The captions provide the transcript; ffmpeg VAD provides the speech segmentation; an Arabic-aware vision-language model OCRs the captions. Output is a HuggingFace `audiofolder` dataset: per-utterance `.wav` clips plus a `metadata.csv` that maps each clip to its transcription. ## Pipeline ``` URLs / profiles -> yt-dlp -> raw video (.mp4) | ffmpeg + silencedetect (VAD) | speech segments (start, end) | ffmpeg -> frames every 0.25 s | OpenCV crop preprocessing | Gemma 4 E2B (via Unsloth Studio) | per-frame Arabic caption text | LLM judge cleanup | trim/dedupe OCR text -> VAD segments | cut audio clips + write metadata ``` A segment is kept only if at least one frame inside its time span was read as Arabic. If the first caption appears only after four blank sampled frames, the whole segment is dropped. Otherwise the audio is cut to the first and last captioned frame window, so trailing video/audio with no on-screen transcription is removed. Consecutive frames with the same OCR text are deduplicated before writing the transcript. Before OCR, the caption crop can be converted to grayscale, sharpened, adaptive thresholded, and cleaned with morphology. After OCR, an optional LLM judge pass receives the rows plus their frame crops and fixes obvious OCR typos, removes repeated words, removes duplicate/near-duplicate rows, and drops title/cover rows. A generic cleanup pass then removes repeated words and collapses adjacent near-duplicates before `ocr.jsonl` is written. A leading title/cover frame is also skipped during alignment when it looks like a longer title containing the next subtitle. After captions have started, OCR stops probing the current speech segment after four consecutive blank frames. If a later speech segment starts with four blank caption frames, OCR assumes captions …