10Accademy-InsightStreamInc/Amharic_LLM_Finetuning

The project aims to enhance NLP capabilities for Amharic Language by developing a data corpus for various NLP applications. The project involves collecting, cleaning, processing data, developing APIs, and automating the pipeline. # Scalable Data Warehouse for LLM Finetuning: API Design for High Throughput Data Ingestion and RAG Retrieval ## Project Overview This projects aims to enhance Natural Language Processing (NLP) capabilities for African languages, focusing on Amharic. This project aims to develop a comprehensive data corpus to support various NLP applications, such as semantic search, content generation, chatbot support, sentiment analysis, and speech recognition. ## Table of Contents - Project Overview - Business Need - Contributors - Tech Stack - Setup Instructions - Usage - Project Structure - Contributing - License ## Business Need The lack of extensive, high-quality text/audio datasets for Amharic is a significant bottleneck for developing competitive NLP products. By collecting and processing a vast amount of text/audio data from diverse online sources, this project will enhance Roots Tech Solutions' ability to create innovative NLP tools for these languages. ## Contributors - Abubeker Shamil - Michael George - Nyamusi Moraa - Eyerusalem Admassu ## Tech Stack - **Programming Languages:** Python, JavaScript (React) - **Web Scraping Tools:** Selenium - **Database:** PostgreSQL - **API Frameworks:** Flask - **Containerization:** Docker, Docker Compose - **Workflow Automation:** Apache Airflow - **Annotation Tool:** Prodigy - **Monitoring:** Grafana ## Setup Instructions ### Prerequisites - Python 3.x - Docker and Docker Compose - PostgreSQL or MongoDB (for local development) ### Installation 1. **Clone the Repository** ```sh git clone github.com cd your-repository ``` 2. **Set Up Virtual Environment** ```sh Copy code python3 -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate ``` 3. **Install Requirements** ```sh pip install -r requirements.txt Set Up Environment Variables ``` 4. **Set Up Virtual Environment** Create a .env file and add the following variables ```env DB_USERNAME='your_username' DB_PAS …