atlasia-ma/Al-Atlas-Moroccan-Darija-Pretraining

Training and data analysis code for Al-Atlas dataset and models # Al-Atlas: Data, Models, Evals ## TL;DR We present a comprehensive dataset for Moroccan darija, addressing the lack of resources for this widely spoken dialect. We detail our collection methodology, provide thorough data analysis, and demonstrate performance improvements in both masked and causal language models after training on this dataset. 1. Try our **demo** showcasing our **AlAtlas LLM** for Moroccan darija. 2. Try our **demo** showcasing our **Masked Language Model** for Moroccan darija. 3. Our **Atlaset** dataset on Hugging Face. 4. **Code** is available on Github. ## 1. Challenges in Moroccan Darija Moroccan darija stands at a fascinating linguistic crossroads, embodying a rich tapestry of influences from Arabic foundations interwoven with Berber, French, and Spanish elements. This vibrant dialect, though widely spoken throughout Morocco, presents unique challenges in the computational linguistics domain. Unlike Modern Standard Arabic (MSA) and other well-resourced languages, Moroccan darija exists in a digital resource vacuum, with **no standardized writing system** to guide its representation. This **orthographic fluidity** where texts may appear in Arabic script, Latin characters, or creative hybridizations, creates significant hurdles for traditional Natural Language Processing (NLP) approaches. The dialect's natural tendency toward **code-switching** and **regional variations** further compounds these challenges, placing Moroccan darija squarely in the category of **under-resourced languages** despite its **cultural significance and widespread use** in daily Moroccan life. These challenges create a significant barrier for developing AI applications for the millions of Moroccan darija speakers, highlighting the **need for dedicated resources** such as datasets, foundational models and NLP tools. ## 2. Data Collection: Atlaset, A Mother Dataset We've assembled what we believe is the most complete collection of publicly available Moroccan darija …