Iambusayor/naijavoices-dataloader

# NaijaVoices & FLEURS Data Loader This repository provides a flexible PyTorch-based data loading pipeline tailored for training speech recognition and text-to-speech (TTS) models on NaijaVoices dataset. It supports both custom datasets and the multilingual Google FLEURS dataset. --- ## 🛠️ Installation 1. **Clone the Repository**: ```bash git clone github.com cd naijavoices-dataloader ``` 2. **Create a Virtual Environment** (optional but recommended): ```bash python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate ``` 3. **Install Dependencies**: ```bash pip install -r requirements.txt ``` *Note: Ensure you have FFmpeg installed for audio processing.* --- ## 📁 Directory Structure ``` naijavoices-dataloader/ ├── dataset.py # Main dataset classes ├── preprocessing.py # Audio and text preprocessing utilities ├── collators.py # Custom data collators for batching ├── requirements.txt # Python dependencies └── README.md # This documentation ``` --- ## 📄 Usage ### 1. **Custom Dataset: NaijaVoices** To load your own dataset: ```python from dataset import NaijaVoices from transformers import Wav2Vec2Processor # Initialize processor (example with Wav2Vec2) processor = Wav2Vec2Processor.from_pretrained('facebook/wav2vec2-base-960h') dataset = NaijaVoices( data_file='path/to/your_dataset.csv', max_audio_len_secs=30, audio_dir='path/to/audio_files', processor=processor, feature_extractor=processor.feature_extractor, tokenizer=processor.tokenizer, language_iso='ha' # Optional: filter by language code ) ``` **CSV File Format**: Ensure your CSV file has the following columns: - `audio_path`: Relative path to the audio file. - `text`: Text label. - `duration`: Duration of the audio in seconds. - `language`: Language code (e.g., 'ha' for Hausa). ### 2. **Google FLEURS Dataset** To load the FLEURS dataset: ```python from dataset import FleursDataset …