elEsquina/LLM-Darija

# DarijaLLM A Language Model for Moroccan Darija (الدارجة المغربية) This project is a personal learning initiative focused on training Large Language Models (LLMs) specifically for Moroccan Darija, a dialect of Arabic spoken in Morocco. The project includes data collection, preprocessing, tokenization, and model training components. ## Project Overview DarijaLLM aims to create a language model that can understand and generate text in Moroccan Darija. The project follows a complete machine learning pipeline from data collection to model training, including web scraping, data cleaning, custom tokenization, and model pretraining. ## Project Structure ### 📁 Root Directory Files - **`environment.yml`** - Conda environment configuration file containing all required Python packages and dependencies for the project - **`README.md`** - This documentation file explaining the project structure and components ### 📁 Jupyter Notebooks - **`WebScrapper.ipynb`** - Web scraping script that collects Moroccan Darija articles from goud.ma website - Scrapes articles by date from 2020 to present - Extracts title, date, content link, and article content - Saves raw data to CSV format - **`DataProcessing.ipynb`** - Data preprocessing and cleaning pipeline - Cleans and normalizes the scraped text data - Removes HTML tags, special characters, and formatting - Prepares data for tokenization and model training - **`Tokenizer.ipynb`** - Custom tokenizer training using Byte Pair Encoding (BPE) - Implements BPE algorithm for Darija text - Trains tokenizer on cleaned Darija corpus - Saves trained tokenizer model - **`Pretraining.ipynb`** - Main model training notebook - Uses HuggingFace transformers library - Trains a language model on the processed Darija data - Implements training with wandb logging for experiment tracking ### 📁 Utils Directory - **`utils/__init__.py`** - Python package initialization file - **`utils/BPE.py`** - Custom Byte Pair Encoding implementation - Complete B …