Iambusayor/naijavoices-dataloader
Domain:
natural language processingRecord type:
software
Creator:
Iam
Host:
# NaijaVoices & FLEURS Data Loader
This repository provides a flexible PyTorch-based data loading pipeline tailored for training speech recognition and text-to-speech (TTS) models on NaijaVoices dataset. It supports both custom datasets and the multilingual Google FLEURS dataset.
---
## 🛠️ Installation
1. **Clone the Repository**:
```bash
git clone github.com
cd naijavoices-dataloader
```
2. **Create a Virtual Environment** (optional but recommended):
```bash
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
```
3. **Install Dependencies**:
```bash
pip install -r requirements.txt
```
*Note: Ensure you have FFmpeg installed for audio processing.*
---
## 📁 Directory Structure
```
naijavoices-dataloader/
├── dataset.py # Main dataset classes
├── preprocessing.py # Audio and text preprocessing utilities
├── collators.py # Custom data collators for batching
├── requirements.txt # Python dependencies
└── README.md # This documentation
```
---
## 📄 Usage
### 1. **Custom Dataset: NaijaVoices**
To load your own dataset:
```python
from dataset import NaijaVoices
from transformers import Wav2Vec2Processor
# Initialize processor (example with Wav2Vec2)
processor = Wav2Vec2Processor.from_pretrained('facebook/wav2vec2-base-960h')
dataset = NaijaVoices(
data_file='path/to/your_dataset.csv',
max_audio_len_secs=30,
audio_dir='path/to/audio_files',
processor=processor,
feature_extractor=processor.feature_extractor,
tokenizer=processor.tokenizer,
language_iso='ha' # Optional: filter by language code
)
```
**CSV File Format**:
Ensure your CSV file has the following columns:
- `audio_path`: Relative path to the audio file.
- `text`: Text label.
- `duration`: Duration of the audio in seconds.
- `language`: Language code (e.g., 'ha' for Hausa).
### 2. **Google FLEURS Dataset**
To load the FLEURS dataset:
```python
from dataset import FleursDataset …