AbdulazeezAde/Yoruba-English-Translation

- # English → Yoruba Machine Translation (fine-tuning afri-mt5-base) **Project goal** Fine-tune a pretrained mT5 model (`masakhane/afri-mt5-base`) to translate English text into Yoruba. This detailed README outlines dataset expectations, preprocessing, training, evaluation, and how to use the modularized code. ## Project layout - `src/` - `data_loader.py` — load and clean TSV/CSV and convert to Hugging Face `Dataset` splits - `preprocess.py` — tokenization and preprocessing helper - `model_init.py` — load tokenizer & model and move model to device - `train.py` — CLI training script using `Trainer` - `evaluate.py` — SacreBLEU evaluation helper - `inference.py` — simple translation helper for examples - `requirements.txt` — dependency list - `English_to_Yoruba_Text_translation.ipynb` — exploratory notebook used to derive scripts ## Dataset format & preprocessing - Expect an input TSV/CSV with columns `English` and `Yoruba`. - Minimal preprocessing: drop missing rows, cast to string, reset index (already implemented in `src/data_loader.py`). - Tokenization: `preprocess_function` creates tokenized inputs/labels using the project tokenizer (default `masakhane/afri-mt5-base`). Default max lengths are 64 tokens for both source and target. ## Dependencies & install Install the required packages into your environment: ```bash pip install -r requirements.txt ``` Key packages: `transformers`, `datasets`, `torch`, `pandas`, `sacrebleu`, `evaluate`, `huggingface_hub`. ## Training Example command (from project root): ```bash python -m src.train --train-file data/train.tsv --output-dir ./results --epochs 2 --batch-size 2 ``` Options: - `--test-file` — path to a held-out test set (if available) - `--model-name` — default `masakhane/afri-mt5-base` - `--push-to-hub` — if set, the trainer will push to Hugging Face hub after training ## Evaluation `src.evaluate.evaluate()` returns a SacreBLEU score for a prepared Hugging Face dataset whose rows contain `English` and `Yo …