ahmedhammami2002/TunisianDialectLLMFinetuning

Tunisian Arabic is a low-resource dialect with significant morphological complexity and multilingual influences. This project evaluates the effectiveness of Parameter-Efficient Fine-Tuning (PEFT) in adapting pre-trained models to this specific linguistic task. # TunisianDialectLLMFinetuning Tunisian Arabic is a low-resource dialect with significant morphological complexity and multilingual influences. This project evaluates the effectiveness of Parameter-Efficient Fine-Tuning (PEFT) in adapting pre-trained models to this specific linguistic task. # Methodology ## Models Evaluated Phi-3-mini-4k-instruct: A ~7B parameter model optimized for efficiency. Qwen2.5-1.5B-Instruct: A smaller, multilingual model with strong native performance in Arabic scripts. ## Optimization Techniques LoRA (Low-Rank Adaptation): Used to freeze pre-trained weights and train small adapter layers, significantly reducing VRAM usage and computation. 4-bit Quantization (bitsandbytes): Both models were loaded in 4-bit to allow for training on consumer-grade hardware without sacrificing significant accuracy. Target Layers: Adapters were applied to attention (Q, K, V, O) and MLP layers. ## Conclusion The results demonstrate that even small models like Qwen2.5-1.5B, when paired with LoRA, can outperform much larger general-purpose models in niche dialectal tasks. While not perfect, the finetuned models successfully shifted from word-for-word translation to context-aware coherence.