wahab-cide/african_languages_llm_project

Training multilingual language models on African languages including Amharic, Fulani, Hausa, Somali, Swahili, and Yoruba. # African Language LLM Project A comprehensive pipeline for training multilingual language models on African languages including Amharic, Fulani, Hausa, Somali, Swahili, and Yoruba. ## 🌍 Project Overview This project aims to develop and train language models specifically for African languages, addressing the underrepresentation of these languages in current NLP research and applications. The pipeline includes data collection, preprocessing, tokenization, and model training with comprehensive logging and evaluation. ### Supported Languages - **Amharic** (am) - Ethiopian Semitic language - **Fulani** (af) - Niger-Congo language family - **Hausa** (ha) - Chadic language family - **Somali** (so) - Cushitic language family - **Swahili** (sw) - Bantu language family - **Yoruba** (yo) - Niger-Congo language family ## 🏗️ Project Structure ``` african_llm_project/ ├── data/ # Data directory │ ├── raw/ # Raw downloaded corpora │ ├── processed/ # Cleaned and processed text files │ └── augmented/ # Data augmentation outputs ├── scripts/ # Data processing scripts │ └── build_tokenized_dataset.py ├── training/ # Training pipeline │ ├── configs/ # Training configurations │ ├── scripts/ # Training and data collection scripts │ └── logs/ # Training logs ├── tokenization/ # Tokenizer models and vocabularies ├── evaluation/ # Model evaluation scripts ├── deployment/ # Model deployment utilities ├── docs/ # Additional documentation └── outputs/ # Trained models and checkpoints ``` ## 🚀 Quick Start ### Prerequisites - Python 3.8+ - Git - DVC (for data versioning) ### Installation 1. **Clone the repository** ```bash git clone github.com cd african_llm_projec …