Dhiadev-tn/darija-translator

A from scratch open-source Tunisian Darija-to-English NLP pipeline. Built from scratch starting on an RTX 3050. # Darija-Translator ### A From-Scratch Tunisian Darija–English NLP Pipeline & Field-Collected Dataset --- ## 🌍 Why This Exists I felt a pang in my chest the day I realized how underrepresented my Tunisian dialect is in the digital world. I searched, and what I found didn't fit: Moroccan datasets that weren't my dialect, and little clean, open Tunisian–English data I could actually build on. So I decided to build my own, openly and from scratch. This project is not just a translation model. It is a statement. Tunisia is a true standing culture that deserves recognition, and it has minds capable of fighting each day despite limited resources and restricted horizons. This is a pipeline built entirely from scratch: no shortcuts, no pre-trained foundations, every component written by hand. What's uncommon here: a small dataset built by hand from real Tunisian speakers, with consent and provenance tracked for every pair. --- ## 🎯 The Problem **Tunisian Darija is spoken by ~12 million people. It has very limited representation in modern NLP research.** - Clean, open, parallel Tunisian Darija–English datasets remain scarce - Existing Arabic NLP tools fail on Tunisian dialect - Arabizi, the way Tunisians write their dialect using Latin letters and numbers (3→ع, 7→ح, 9→ق, 5→خ), is completely unsupported by standard tokenizers - The digital world has ignored an entire identity This project is a step toward changing that. --- ## 🛠️ What Was Built A complete NLP pipeline built from scratch, starting on an RTX 3050 Laptop (4GB VRAM): ``` vocab.py → Custom BPE tokenizer with Arabizi support Handles 3, 7, 9, 5 as protected markers 16,000 token vocabulary data_loader.py → Dataset pipeline 35,977 cleaned Darija-English pairs clean_data.py → Data cleaning pipeline Removed duplicates, noise, and Moroccan-specific terms model.py → Nano-Transformer architecture 15.6M parameters Built entirely in PyTorch train.py → Full train …