NathanaelTamirat/tiny-amharic-GPT

Amharic text generation model ## Amharic Language Text Generation Model This project implements a GPT(Generatively Pretrained Transformer) form the popular paper by google "Attention is all you need" model for text generation using an Amharic language corpus. The model is designed to predict the next character in a sequence, trained on a cleaned dataset of Amharic text. Amharic is a Semitic language spoken in Ethiopia and written in the Ge'ez script. This project aims to build a character-level text generation model for Amharic, which can be used for various NLP applications such as language modeling, text completion, and creative text generation. ### Features - Character-level text generation for the Amharic language. - Trained on a cleaned and preprocessed Amharic text corpus. - Utilizes modern deep learning techniques for sequence prediction. ### Dataset The dataset consists of a large collection 7GB of Amharic text, cleaned and preprocessed to remove noise and irrelevant content. The data is split into training and validation sets to evaluate the model's performance. ### installation 1. Clone the repo ``` git clone git@github.com:NathanaelTamirat/tiny-amharic-GPT.git cd tiny-amharic-GPT ``` 2. Create and activate a virtual environment: ``` python -m venv venv #linux source venv/bin/activate # On Windows, use venv\Scripts\activate ``` 3. install the required packages: ``` pip install -r requirements.txt ``` ### Training Training the Amharic text generation model on a CPU would take significantly longer and is not recommended. Instead using GPU is recommended for efficiency. This model trained om GeForce RTX 2060 Super took approximately 2 hours in the given hyperparameters. i. Preprocess the dataset: remove noise and irrelevant content. ii. The training script (tiny_GPT.py) includes the following steps: 1. Model Building: Define and compile the RNN model. 2. Data Preprocessing: Load and clean the dataset, then convert it to sequences of characters. 3. Training: Train the mod …