ismailousa/fulfulde-translator

High-quality Fulfulde translations using NLP # Fulfulde Translator > Fine-tuning NLLB & M2M100 models for Fulfulde ↔ English/French translation ## 🌍 Overview This project focuses on fine-tuning multilingual translation models to improve translation quality between Fulfulde (ff), English (en), and French (fr), with a particular focus on the Fulfulde dialect. The implementation uses Parameter-Efficient Fine-Tuning (PEFT/LoRA) for training efficiency and provides tools for model distillation to create smaller, deployment-ready models. ## 🚀 Getting Started ### Prerequisites - Python 3.12+ - PyTorch 2.6.0+ - Transformers 4.48.3+ - PEFT 0.7.1+ ### Installation 1. Clone this repository: ```bash git clone github.com cd fulfulde-translator ``` 2. Install dependencies: ```bash # Using pip pip install -e . # Or using Poetry poetry install ``` ## 🧠 Project Structure ``` fulfulde-translator/ ├── data/ # Training and evaluation data │ └── adamawa_english_fulfulde_french_fub.jsonl ├── src/ # Source code │ ├── config/ # Configuration files │ ├── models/ # Model utilities │ ├── tokenization/ # Tokenizer utilities │ ├── data/ # Dataset processing │ ├── training/ # Training code │ ├── evaluation/ # Evaluation metrics │ └── distillation/ # Model distillation ├── scripts/ # Shell scripts for training ├── examples/ # Example usage scripts ├── tests/ # Unit tests └── README.md # This file ``` ## 📊 Data Format The training data is provided in `.jsonl` format with the following structure: ```json { "english": "Buuba is Fulani.", "fulfulde": "Buuba pullo.", "french": "Buuba est Peul." } ``` The code automatically converts this to the Hugging Face translation format: ```json { "translation": { " …