Kirouane-Ayoub/Darija-GPT

# Build and Train GPT2 Model from Scratch : ## Overview Welcome to the **Darija-GPT** project, where we aim to train a **GPT-2 model** specifically for generating text in Algerian Darija. Algerian Darija is a spoken dialect with limited written resources, making it a unique and challenging language to model. GPT-2 is renowned for its text generation capabilities and has been widely used across various applications. By training GPT-2 on Algerian Darija, we hope to create a model that can understand and generate text in this dialect, despite the limited availability of written data. The project Details can be found in the This blog post ## Dataset: Algerian-Darija ### Introducing the Algerian-Darija Dataset I have compiled a new dataset named **Algerian-Darija**. This dataset contains text in **🇩🇿Algerian Darija**, collected from a variety of sources, including existing datasets on Hugging Face, web scraping, and YouTube comments and transcript APIs. The **v1** split consists of more than **170,000 rows** of split and partially cleaned text. ### Sources : **The text data was gathered from:** - **Hugging Face Datasets:** Pre-existing datasets relevant to Algerian Darija. - **Web Scraping:** Content from various online sources, ensuring a diverse range of topics and contexts. - **YouTube API:** Transcriptions from Algerian Darija videos and comments on YouTube, capturing conversational and colloquial speech. **Note:** Some text data from the YouTube Transcript API may contain imperfections due to limitations in speech-to-text technology for Algerian Darija. Additionally, the dataset still requires further cleaning to improve its quality for more advanced NLP tasks. This dataset forms the foundation for our project, Darija-GPT, which aims to train a GPT-2 model specifically on Algerian Darija, harnessing this unique linguistic resource to build a robust and effective language model. ## Training ##### Prerequisites 1. **Training Data**: Ensure your raw text d …