Rachelslh/algerian-dialect-GPT

Training a GPT on the Algerian dialect corpus from scratch using MPS device. # Algerian Dialect GPT This repository contains the implementation of a GPT model pre-trained on the Algerian dialect. The project aims to contribute to the underrepresented area of natural language processing (NLP) for dialects, with a focus on generating coherent and contextually relevant text in Algerian Arabic. ## Table of Contents - Model - Dataset - Training - Results - Usage - Contributing ## Model architecture This project uses a transformer decoder, pre-trained on a corpus of Algerian dialect text. The model is trained to generate text that is contextually relevant and adheres to the linguistic characteristics of the Algerian dialect. Tried to grasp the most of GPT2/GPT3 model settings in order to mimic those models, although i've had to set some of the pramaters to lower values given the fact that i'm training on MPS device. ### Config ```python vocab_size: 10240 # The tokenizer uses 10000 tokens only, but this is a better number because it's divisble by many powers of 2 embedding_dim: 768 n_layers: 12 heads: 12 head_size: 64 block_size: 256 ``` ### Additional info - Used pre-layer normalization as demonstarted by this paper instead of post-layer normalization as in the "Attention is all you need" paper. - Tried adding dropout layers after every fully connected layer, it did not enhance training in this case, i also did not have any case of overfitting, hence the use of dropout was not ideal here. - Just like in the GPT-2 paper, weights are being shared here between the embedding layer and the final linear layer. - I am using Flash attention here for optimization purposes, there is a coded (but commented out) version of the self attention mechanism in model.py, put together with the help of Andrej Karpathy's videos. ## Dataset I have used a huggingface dataset, contains over 170k rows of the Algerian dilaect with each row contains a different number of tokens. The latter contains several french words and since we're taking baby steps here, …