jadmassu/amharic_llm_finetuning

This project enhances NLP capabilities for Amharic by developing a comprehensive dataset to support semantic search, content generation, chatbots and sentiment analysis # Scalable Data Warehouse for LLM Finetuning: API Design for High Throughput Data Ingestion and RAG Retrieval ## Project Overview This projects aims to enhance Natural Language Processing (NLP) capabilities for African languages, focusing on Amharic. This project aims to develop a comprehensive data corpus to support various NLP applications, such as semantic search, content generation, chatbot support, sentiment analysis, and speech recognition. ## Table of Contents - Project Overview - Business Need - Setup Instructions - Project Structure - License ## Business Need The lack of extensive, high-quality text/audio datasets for Amharic is a significant bottleneck for developing competitive NLP products. By collecting and processing a vast amount of text/audio data from diverse online sources, this project will enhance Roots Tech Solutions' ability to create innovative NLP tools for these languages. ## Setup Instructions ### Prerequisites - Python 3.x - conda ### Installation 1. **Clone the Repository** ```sh git clone git@github.com:jadmassu/Amharic_LLM_Finetuning.git cd Amharic_LLM_Finetuning ``` 2. **Install Requirements** ```sh pip install -r requirements.txt Set Up Environment Variables ``` 3. **Start Notebook** ```sh conda activate base jupyter lab ``` ## Project Structure ├── notebooks # jupyter notebooks for interactive data analysis and exploration ├── data # Contains datasets and raw data files used in the project ├── scripts # Python scripts for implementing project functionality ├── requirements.txt # Lists Python dependencies required for the project. ├── README.md # Provides documentation and instructions for understanding and using the project └── ... ### License This project is licensed under the MIT License. See the LICENSE file for details.