HafsatiMohammed/darija-asr-diarization-metadata

Generate speaker, overlap, and ASR usability metadata for Moroccan Darija speech datasets using `pyannote.audio`. # Darija ASR Diarization Metadata Generate speaker, overlap, and ASR usability metadata for Moroccan Darija speech datasets using `pyannote.audio`. This project enriches an existing speech dataset with metadata that helps identify: - how many speakers are present in each sample - whether speakers overlap - how dominant one speaker is - how fragmented the conversation is, - and whether a sample is likely suitable for standard single-speaker ASR training. ## Why this project exists Many ASR datasets contain samples that are not equally useful for training a standard speech-to-text model. Some clips are clean single-speaker utterances. Others contain: - multiple speakers, - turn-taking dialogue, - overlapping speech, - noisy diarization structure, - or mixed conversational audio that may not match a single transcript cleanly. Instead of removing everything manually, this project adds metadata automatically so you can: - filter clean samples - keep difficult samples but tag them - build cleaner train/validation/test sets - analyze corpus quality - and study the effect of multi-speaker speech on ASR training. ## Main idea The pipeline: 1. loads a Hugging Face dataset 2. runs speaker diarization with `pyannote/speaker-diarization-community-1` 3. computes speaker and overlap statistics 4. derives a heuristic ASR usability score 5. adds new metadata columns 6. merges and reshuffles the data 7. creates new train/validation/test splits optional you can remove 8. pushes the final dataset back to the Hugging Face Hub ## Supported source datasets Examples in the script: - `adiren7/darija_speech_to_text` - `KandirResearch/DarijaTTS-clean` You can add you own dataset, you may need to change the columns name ## Generated metadata For each audio sample, the script adds: - `duration_sec` - `num_speakers` - `multiple_speakers` - `speaker_turns` - `turns_per_minute` - `dominant_speaker_ratio` - `second_speaker_ratio` - `non_dominant_speech_ratio` - `speaker_balance_score` …