KLiFF2606/The-Moroccan-Darija-Word2Vec-Project

The Moroccan Darija Word Embedding project. # The-Moroccan-Darija-Word2Vec-Project (Lire ce document en français) The Moroccan Darija Word Embedding project is an open-source project that aims to create a large word embedding database for Moroccan Darija. (You don't know what is a word embedding database? No problem you can find all you need here: en.wikipedia.org) ## Why do we build such a component? Moroccan Darija is being more and more used by moroccan Internet users. Writing tweets in Darija is now a trend and will soon become the rule. Many pages and groups on facebook are increasignly using Darija as their main writing language. Finally, almost all WhatsApp conversations are written in Darija using arabic letters. The usage of mobile apps allows more ease of use with the arabic keyword available on almost every smartphone. Therefore, it becomes obvious that building an effective NLP algorithm for moroccan Internet users has to target Darija. Word embedding (word to vector for instance) can be considered as the first building stone to any successfful NLP algorithm. It allows machine learning algorithms to input a word representation that reflects a learned pseudo-meaning. This is proven to remarquably improve the performance of algorithms used for tasks like sentiment analysis, entity detection, etc.. For many languages, word to vector databases have already been built and are available for free on Internet. Some of them contain billions of tokens. Also, a database already exists for Classical Arabic and none exists yet for Moroccan Darija. For any NLP professional, researcher or student, the lack of such a database is simply a concrete wall ahead of any NLP project. For the industry, it means a whole set of capabilities to forget. For us, the lack of such a database is an opportunity to gather a new moroccan generation of AI enthousiats arround a common project and get all the benefits from building it. ### A note about The Darija challenge With the increasing inte …