Kirouane-Ayoub/Mistral-7b-Meets-Darija

# Mistral 7b Meets Darija 🇩🇿: A Continual Pre-training Journey Welcome to the **Mistral 7b Meets Darija 🇩🇿** project! This project aims to develop a **large language model** capable of **understanding** and **generating text** in **Algerian Darija**. We're using **continual pre-training** to teach the model **Arabic** and **French** first, and then **fine-tuning** it on a specifically curated **Algerian Darija dataset**. #### Technical Approach: - **Model**: We're primarily using **Mistral-7B**, a powerful and efficient large language model. - **Training Method**: Continual pre-training on Arabic and French datasets followed by fine-tuning on Algerian Darija data. - **Tools and Techniques:** Leveraging **PEFT**, **Unsloth Framework**, **RoPE**, and **Flash Attention-2** for optimized training and performance. The project Details can be found in the This blog post . ## Running the train Script : before runing the train script, You should install the requirements: - **For Pytorch 2.1.0** : ``` pip install --upgrade --force-reinstall --no-cache-dir torch==2.1.0 triton \ --index-url download.pytorch.org pip install "unsloth[cu118] @ git+github.com" pip install "unsloth[cu121] @ git+github.com" pip install "unsloth[cu118-ampere] @ git+github.com" pip install "unsloth[cu121-ampere] @ git+github.com" ``` - **For Pytorch 2.1.1:** ``` pip install --upgrade --force-reinstall --no-cache-dir torch==2.1.1 triton \ --index-url download.pytorch.org pip install "unsloth[cu118-torch211] @ git+github.com" pip install "unsloth[cu121-torch211] @ git+github.com" pip install "unsloth[cu118-ampere-torch211] @ git+github.com" pip install "unsloth[cu121-ampere-torch211] @ git+github.com" ``` - **For Pytorch …