silomalo/luhya-transformer

# English ↔ Luhya Translator This project fine-tunes a multilingual sequence-to-sequence model to translate between English and Luhya in both directions. ## Features - Train on the `mamakobe/luhya-multilingual-dataset` Hugging Face dataset by default. - Cache Hugging Face datasets and base models in project `data/` for faster repeated runs. - Automatically choose safer training settings for CUDA, Apple Silicon MPS, or CPU. - Translate English → Luhya from the command line. - Translate Luhya → English from the command line. - Use a Gradio browser interface for interactive translation. - Run the workflow from `notebook.py`. ## Requirements Install the Python dependencies: ```bash pip install -r requirements.txt ``` If you have an NVIDIA GPU, verify that PyTorch can see it: ```bash python -c "import torch; print(torch.cuda.is_available()); print(torch.cuda.get_device_name(0) if torch.cuda.is_available() else 'no cuda')" ``` ## Dataset By default, training uses the Hugging Face dataset: ```txt mamakobe/luhya-multilingual-dataset ``` The script automatically reads the dataset columns: - `english_text` - `luhya_text` and normalizes them internally for bidirectional training. Rows with missing or empty English/Luhya text are ignored during training. ### Local cache On first run, the Hugging Face dataset and base model are cached in: ```txt data/ ``` This project-local cache avoids downloading the same files again on repeated runs from this project. You can delete `data/` if you want to force a fresh download. ### Optional local CSV You can also train from a local CSV file. The local file must contain these columns: ```csv english,luhya How are you?, Thank you, ``` ## Train the model Train with the default Hugging Face dataset: ```bash python notebook.py --mode train ``` Train with a local CSV instead: ```bash python notebook.py --mode train --dataset luhya_english_dataset.csv ``` The trained model and tokenizer are saved to: ```txt luhya_englis …