tpchikumbu/GPTiny

Predictive text generation for African languages using transformer architecture # GPTiny GPTiny is an African language, transformer based, predictive text generation model. It was inspired by the ability to simplify training and implementation of famous generative pre-trained transformer (GPT) models. Other projects that perform similar tasks include minGPT and nanoGPT. A single Jupyter notebook contains all code for the projects. This allows it to be run on cloud computing platforms with access to powerful GPUs like Google Colabs. Several datasets are included with the training file. These come from the NCHLT monolingual corpora released by the South African Centre for Digital Language Resources (SADiLaR). The data has already been split into training, validation and test sets and basic cleaning has been performed. Most testing was performed using the Ndebele dataset. A report on the models performance is included within the repository.