kiplangatkorir/swahili-gpt2-distilled

A distilled version of the GPT-2 model, fine-tuned specifically for Swahili language tasks. # Swahili GPT-2 Distilled This repository hosts a distilled version of the GPT-2 model fine-tuned on a custom Swahili language corpus. The project aims to provide an efficient, smaller language model capable of generating Swahili text and performing other natural language processing (NLP) tasks. This model can be used in various applications including chatbots, content generation, and language modeling, especially in resource-constrained environments. ## Project Motivation Swahili is spoken by over 100 million people, predominantly in East Africa. Despite its wide usage, Swahili has been underrepresented in state-of-the-art AI models, particularly in generative language modeling. The aim of this project is to bridge this gap by training a compact, efficient model that maintains the richness of the Swahili language while being computationally accessible. By distilling a GPT-2 model, we seek to: - Provide a lighter alternative to large-scale models for Swahili. - Enable Swahili NLP applications in resource-constrained environments (e.g., mobile, embedded systems). - Create a foundational model that can be expanded to other African languages. ## Key Features - **Distilled GPT-2 Model**: A compact version of GPT-2 optimized for efficiency while retaining the core language generation capabilities. - **Swahili-Centric Training**: Fine-tuned specifically on Swahili text from various domains (news, blogs, educational material). - **Resource-Efficient**: The model is optimized for deployment in environments with limited computational resources (e.g., mobile devices). - **Open Access**: The model and code are available for free, encouraging collaboration and expansion to other low-resource African languages. ## Table of Contents 1. Installation 2. Data Preparation 3. Training 4. Evaluation 5. Usage 6. Contributing 7. License 8. Acknowledgements 9. References ## Installation ### 1. Clone the Repository ```bash git clone github.com …