Yosef-Ali/amharic-llm-data

# Amharic LLM Data Collection Pipeline ### 🌍 A Modern, Reproducible Approach for Low-Resource Language Models > **For Researchers & Teams**: This pipeline demonstrates how to build high-quality instruction datasets for low-resource languages, specifically Amharic. The approach is reproducible for any language. ## 📋 Table of Contents - Overview - Key Innovations - Quick Start - Architecture - Data Sources - Reproducing for Other Languages - Handling Modern Challenges - Results & Benchmarks - Contributing - Citations ## Overview This project implements a production-ready data collection pipeline for Amharic LLMs, combining approaches from: - **Walia-LLM** (task-specific dataset conversion) - **Modern practices** (synthetic generation, Parquet format) - **2025 standards** (quality filtering, deduplication) ### 🎯 Problem Solved Low-resource languages face three main challenges: 1. **Lack of instruction data** - Few native instruction-following datasets 2. **Quality issues** - Machine translation introduces artifacts 3. **Technical barriers** - Complex setup and deprecated formats Our pipeline addresses all three with an automated, quality-focused approach. ## Key Innovations ### 1. Multi-Source Strategy ```python sources = { 'structured': ['AfriSenti', 'MasakhaNews'], # Existing NLP datasets 'generative': ['WikiMezmur', 'Folktales'], # Cultural content 'synthetic': ['GPT-4', 'Claude'], # High-quality generation 'web': ['BBC', 'VOA', 'DW'] # Fresh content } ``` ### 2. Quality-First Approach - **Amharic character ratio checking** (>70% native content) - **Deduplication** at multiple levels - **Template diversity** (5-14 per task) - **Automatic filtering** for length, repetition, toxicity ### 3. Modern Technical Stack - **Parquet format** for 10x faster loading - **Streaming support** for large datasets - **QLoRA training** for consumer GPUs - **Automated pipeline** with error recovery ## Quick Start ### Prerequisi …