Achraf-Aouam/darija_embedding

# Darija Word Embeddings Project This project focuses on learning high-quality word embeddings for Moroccan Darija using a custom-curated corpus. The process involves extensive text preprocessing, subword tokenization using Byte Pair Encoding (BPE), and training Word2Vec (Skip-gram with Negative Sampling) models. ## Project Goal To create robust and semantically rich vector representations for Darija words and subwords, which can be used for various downstream Natural Language Processing (NLP) tasks such as sentiment analysis, machine translation, text classification, and information retrieval specific to the Darija dialect. ## Corpus The primary corpus consists of a collection of `.txt` files sourced from various websites, containing a mix of Darija (written in Arabic or Latin/Arabizi script), Standard Arabic, and some French. The total size of the raw text data is approximately 1 GB, containing around 600 million characters. **Note on Corpus Curation:** The "good_articles" portion of the initially considered corpus was excluded from the final training set. This decision was made because these articles predominantly contained Modern Standard Arabic (MSA) rather than Moroccan Darija, and the aim was to create embeddings specifically tailored to Darija. ## Project Architecture & Process The project is structured into three main stages, implemented across different Jupyter Notebooks: 1. **Data Preprocessing (`prep_ar_custom.ipynb`)** * **Objective:** To clean and normalize the raw text data, transforming it into a suitable format for tokenization and model training. This step is crucial for handling the diverse and often noisy nature of web-scraped Darija. * **Key Steps:** 1. **Corpus Loading:** Reading all `.txt` files from the data directory. 2. **Initial Cleaning:** Removing noise such as URLs, email addresses, HTML tags, and irrelevant markup. 3. **French Line Detection & Filtering:** Identifying and removing lines that are predominantly French, w …