khalidAgnaber/Darija-Word-Embeddings

Developed a Word2Vec embedding model for Darija to capture semantic relationships between words. The project involved extensive text preprocessing, including Arabic text cleaning, tokenization, stop-word removal, and sentence filtering, followed by training a skip-gram Word2Vec model with negative sampling. # Darija Word2Vec Model - Documentation ## Overview This code builds a Word2Vec embedding model specifically for Darija (Moroccan Arabic dialect). The system processes Arabic text data, cleans and tokenizes it, then trains a skip-gram Word2Vec model with negative sampling to create dense vector representations of Darija words. These embeddings capture semantic relationships between words in the dialect. ## Data Preprocessing and Text Cleaning - **Arabic Text Processing:** - Removes Arabic diacritics (harakat) like Fatha, Damma, Kasra, etc. - Filters out non-Arabic characters, emojis, punctuation, and numbers. - Applies a threshold check (ARABIC_THRESHOLD = 0.5) to ensure text is predominantly Arabic. - Normalizes spaces and removes extra whitespace. - **Sentence Processing:** - Splits text into sentences (using periods as delimiters). - Filters sentences by minimum length (MIN_SENTENCE_LENGTH = 20). - Requires a minimum token count per sentence (MIN_TOKENS_PER_SENTENCE = 5). - **Tokenization:** - Implements custom arabic_tokenize() function to split text into words. - Removes very short tokens (minimum length: MIN_WORD_LENGTH = 3). - Filters out Darija stop words like "في", "من", "على", etc. ## Corpus Handling - **Data Organization:** - Reads text files from the DATA_PATH directory. - Maintains source information (category/filename) for each processed sentence. - Saves processed data to CSV for potential reuse. - **Data Filtering:** - Skips non-Arabic content. - Requires a minimum of 3 valid sentences per file. ## Word2Vec Model Architecture - **Model Configuration:** - Embedding Dimension: 300 - Context Window: 2 - Architecture: Skip-gram model - Negative Sampling: 10 samples - Learning Rate: Starting at 0.025, minimum 0.0001 with decay - Training Epochs: 10 - (We tried different context windows, epochs and learning rates) - **Training Process:** - Custom LossLogger callback tracks and reports model loss after each epoch. - Multi-threaded training …