isrish/AVSpeech_Amharic

# AVSpeech-Style Amharic Curation Pipeline This repository provides a **reproducible Python pipeline** to curate an AVSpeech-style dataset for **Amharic (am)** from YouTube: - Discover Amharic videos - Download audio/video assets - Detect speech with VAD and remove music - (Optional) Verify Amharic language via Whisper - (Optional) Transcribe segments (Amharic) with Whisper - Export a JSONL manifest and clip files (3–10s) > ⚠️ This is an independent, open-source baseline inspired by the high-level AVSpeech idea (audio-visual, speech-focused segments). It does **not** replicate the original internal pipeline. --- ## Quick Start ### 1) Install system deps - **ffmpeg** (required) - (GPU optional) CUDA/cuDNN if using Whisper / Torch acceleration On Ubuntu: ```bash sudo apt-get update && sudo apt-get install -y ffmpeg ``` ### 2) Create & activate a virtual env ```bash python3 -m venv .venv source .venv/bin/activate ``` ### 3) Install Python dependencies ```bash pip install -r requirements.txt ``` ### 4) Configure Copy the sample config and edit as needed: ```bash cp configs/sample_config.yaml configs/config.yaml ``` Key fields you may want to change: - `output_root`: where to write downloads, clips, and manifests - `youtube.query_terms`: search keywords used to discover Amharic content - `whisper.enabled`: enable/disable Whisper transcription & language verification ### 5) Run the pipeline Search → Download → Curate → Export: ```bash # Search for candidate videos (stores JSON index) python -m avspeech_amharic.cli search --max-results 50 # Download videos & extract audio python -m avspeech_amharic.cli download # Curate (VAD, music removal, language verify, segment into 3–10s) python -m avspeech_amharic.cli curate # Export a JSONL manifest python -m avspeech_amharic.cli manifest ``` The final manifest will be in `manifests/dataset.jsonl`. Each line looks like: ```json { "video_id": "abc123", "segment_id": "abc123_000123_000129", "start": 123.0, "end": 129.2 …