Mehdi192002/Darija-Transliteration

Darija transliteration # Darija Transliteration System A comprehensive machine learning pipeline for bidirectional transliteration between Moroccan Darija (Arabic script) and Latin script (Arabizi), built using transformer-based models. ## 📋 Overview This project provides a complete end-to-end solution for: - **Dataset Generation**: Converting Arabizi to Arabic script using LLM-powered transliteration - **Model Training**: Fine-tuning ByT5 models for accurate bidirectional transliteration - **Inference**: Real-time transliteration with intelligent word-level processing ## 🚀 Quick Start ### Prerequisites ```bash pip install pandas transformers datasets torch scikit-learn google-generativeai ``` ### 1. clean Dataset (Optional - datasets provided) ```bash # Refine and filter the dataset python clean_and_refine.py python filter_non_darija.py ``` ### 2. Train the Model ```bash # Stage 1: Initial training python train_model.py # Stage 2: Generate synthetic data python generate_fake_words.py # Stage 3: Fine-tune with augmented data python finetune_model.py ``` ### 3. Use the Model ```bash # Interactive transliteration python use_model.py ``` **Example Usage:** ``` Arabic Input: كيف داير خويا؟ Latin Output: kif dayer khoya? Arabic Input: واش بغيتي تمشي معايا؟ Latin Output: wach bghiti tmchi m3aya? ``` ## 📊 Dataset Statistics | Dataset | Rows | Description | |---------|------|-------------| | Raw Comments | ~1,000 | Social media comments (Instagram Reels) | | Cleaned Robust | 863 | Validated Darija sentences | | Final Dataset | 808 | Quality-filtered pairs | | Word Pairs | ~2,000 | Word-level alignments | | Synthetic | 5,000 | Rule-based generated pairs | | Augmented | ~6,000 | Combined training set | ## 🔧 Configuration ### API Keys The project uses Google's Gemini API for dataset generation. Add your API key in: - `generate_dataset.py` - `clean_and_refine.py` - `pair_words.py` ```python API_KEY = 'YOUR_API_KEY_HERE' ``` ### Model Selection - **Base Model**: `google/byt5-smal …