chaima-Khenafif03/darija-latin-to-arabic-translator
Domaine:
natural language processingType de record:
model
Créateur:
cha
Hôte:
Fine-tuned mBART-50 model that translates Algerian Darija from Latin script to Arabic script, handling telecom vocabulary and spelling variation
# Darija Latin → Darija Arabic Translator
A fine-tuned mBART-50 model that translates Algerian Darija written in Latin script into Darija written in Arabic script — handling French/English loanwords, telecom-specific terms (4G, GB, DA...), and the many different Latin spellings people use for the same word.
## Example
| Input (Darija Latin) | Output (Darija Arabic) |
|---|---|
| `3andi offre tae 300DA w cnx lente` | `عندي عرض تاع 300دج وكونكسيون بطيئة` |
| `salam 3alaykom` | `السلام عليكم` |
| `wash kayen men les offres jdida?` | `واش كاين من العروض الجديدة؟` |
**Handles spelling variation**: the dataset includes multiple real-world spellings of the same word, so the model generalizes across them rather than requiring one "correct" transliteration:
```
le téléphone te'i ysserkouli les données personnelles
le téléphune ta3y yssar9oly les dunnées persunnelles
le téléphone ta3i yssar9ouli les données personnelles
```
→ all map to: `التيليفون تاعي يسرقولي البيانات الشخصية`
## Results
Fine-tuned `facebook/mbart-large-50-many-to-many-mmt` on a custom 10,000-sample Darija Latin/Arabic parallel dataset, with custom telecom tokens added to the vocabulary (3G, 4G, 5G, GB, DA, Mbps, lte, adls).
| Metric | Score |
|---|---|
| BLEU | 0.83 |
## Project structure
```
├── data/ # sample of the parallel dataset (full 10K-row dataset not published)
├── notebooks/ # original development notebook
├── src/
│ ├── preprocess.py # dataset loading, tokenization, special tokens
│ ├── train.py # fine-tuning script
│ └── predict.py # inference
├── examples/ # sample inputs/outputs
└── requirements.txt
```
## Usage
```bash
pip install -r requirements.txt
# Train
python src/train.py path/to/darija_dataset_updated.csv
# Translate (automatically downloads the model from Hugging Face Hub on first run)
python src/predict.py "3andi offre tae 300DA w cnx lente"
```
## Dataset
Custom parallel dataset of 10,000 Darija Latin → Darija Arabic se …