thiomajid/hausa_lm

Language Modeling for Hausa # Hausa Language Models ## Overview This repository contains research and implementations of language models for Hausa, one of Africa's major languages spoken primarily in northern Nigeria, Niger, and other parts of West Africa. The project aims to advance natural language processing capabilities for Hausa by developing specialized language models for various tasks. ## Objectives - Develop pre-trained language models for Hausa - Fine-tune models for specific NLP tasks - Improve accessibility of NLP tools for the Hausa-speaking community - Bridge the technological gap for low-resource languages ## Tasks The models in this repository are being developed for various NLP/Vision tasks, including but not limited to: - Text generation - Machine translation - Vision question answering ## Motivation Despite being spoken by over 70 million people, Hausa remains underrepresented in current NLP research and applications. This project aims to address this disparity by creating resources that can be used in practical applications and further research. ## Contributing Contributions are welcome! Whether you're a native Hausa speaker, ML practitioner, or NLP researcher, your input can help improve these resources. ## Installation To set up the development environment for this project: 1. Create a virtual environment using `uv`: ```bash uv venv ``` 2. Activate the virtual environment: ```bash # On Unix/Linux/macOS source .venv/bin/activate # On Windows .venv\Scripts\activate ``` 3. Install dependencies: ```bash uv pip install pyproject.toml -e . ``` ## License This project is licensed under the Apache License 2.0 - see the LICENSE file for details. ## Scripts Here you can include various scripts related to the development and fine-tuning of the Hausa language models. This may include pre-processing scripts, training scripts, and any utility functions that are helpful for working with the data or models. - To train a custom tokenizer ```bash python3 train_token …