nyacly/runyoro-llm-data-pipeline

This is a data processing pipeline for training an LLM # Runyoro/Rutooro LLM Data Pipeline This repository contains a data processing and training pipeline for developing language models for Runyoro/Rutooro, an endangered Bantu language spoken in Uganda. ## Issues Fixed The following issues have been identified and resolved in this refactored version: ### 1. Tokenizer Issues - **Problem**: The original code attempted to use fast tokenizers with mT5, which caused tiktoken-related errors - **Solution**: Modified `tokenizer_utils.py` to use the slow T5Tokenizer and skip custom tokenizer training for SentencePiece models ### 2. Mixed Precision Configuration - **Problem**: Mixed precision was hardcoded to be disabled, preventing GPU acceleration - **Solution**: Implemented proper CUDA detection and mixed precision configuration based on hardware availability ### 3. Dependencies - **Problem**: Missing dependencies and conflicting package versions - **Solution**: Updated `requirements.txt` with proper dependencies including `sentencepiece` and `accelerate` ### 4. Data Processing - **Problem**: Text iterator yielded entire files instead of individual lines - **Solution**: Modified `_text_iterator` to yield individual lines for better tokenization ## Installation 1. Clone the repository: ```bash git clone github.com cd runyoro-llm-data-pipeline ``` 2. Install dependencies: ```bash pip install -r requirements.txt ``` 3. (Optional) Configure accelerate for distributed training: ```bash accelerate config ``` ## Usage ### Data Processing Place your text data in the `processed_data/processed_text/` directory as `.txt` files. ### Training Run the training script: ```bash python3 -m scripts.train_llm \ --processed_data_path processed_data/processed_text \ --model_name google/mt5-small \ --output_dir ./models/runyoro_llm_model \ --tokenizer_dir ./tokenizer \ --checkpoint_dir /tmp/runyoro_checkpoints \ --mixed_precision no \ --load_in_8bit \ --num_train_epochs 5 \ --per_device_tr …