OthmanSALAHI/script-switching-darija

This project investigates the vulnerability of state-of-the-art Arabic transformer models to "script-switching"—the practice of writing Arabic dialects (like Moroccan Darija) using Latin characters and numbers (Arabizi). By treating transliteration as a zero-effort adversarial attack, we demonstrate that models like AraBERT and DarijaBERT suffe # Evaluating Script-Switching Robustness 🛡️ ### Robustness Auditing of Arabic Transformer Models against Arabizi Adversarial Evasion --- ## 📌 Overview This repository contains the code and white paper for a controlled robustness audit of Arabic transformer models in the context of Moroccan misinformation detection. We investigate a critical and largely unexplored vulnerability: **Script-Switching**. Millions of Moroccan social media users write in **Arabizi** (Arabic phonemes using Latin characters/numbers). While AI systems are highly accurate on Arabic script, they are often "blind" to Arabizi. This study demonstrates that script-switching acts as a **zero-effort adversarial attack** that can bypass state-of-the-art detection systems with no technical expertise. > **The Core Problem:** Most Arabic NLP models are trained and tested **exclusively on Arabic script**, creating a massive security gap on platforms where Arabizi is the dominant vernacular. --- ## 🔬 Research Findings: The Robustness Paradox Our study reveals a counterintuitive reality: **Higher benchmark accuracy does not predict greater operational reliability.** | Model | Arabic F1 (Baseline) | Arabizi F1 (Adversarial) | Performance Drop | |---|---|---|---| | **DarijaBERT** | 0.7840 | 0.4979 | **−36.5%** | | **AraBERT-Twitter** | 0.8725 | 0.3406 | **−60.9%** | ### 📊 Visual Analysis | Model | Baseline (Arabic) | Adversarial (Arabizi) | |---|---|---| | **DarijaBERT** | | | | **AraBERT-Twitter** | | | **Key Finding:** AraBERT-Twitter — the "stronger" model on standard Arabic — failed significantly harder on Arabizi, misclassifying **99%** of real news articles as fake. --- ## 🏗️ Project Structure ``` code-switching/ ├── data/ │ ├── train.csv # 16,000 training samples │ ├── val.csv # 2,000 validation samples │ ├── test.csv # 2,000 test samples (Arabic script) │ └── test_arabizi.csv # 2,000 test samples (Arabizi — adversarial …