Eliphaz21/FULL-finetuning-amharic-language-in-agricutlture

# 🌾 Amharic LLM Fine-Tuning Pipeline for Agriculture A complete end-to-end notebook for fine-tuning open-source Large Language Models on Amharic agricultural datasets using QLoRA and Unsloth. This notebook is designed for researchers, students, startups, and developers who want to build high-quality Amharic AI assistants without creating an entire training pipeline from scratch. Simply prepare your dataset, run the notebook, and deploy your model. --- # ✨ Features ✅ Google Sheets dataset loading ✅ Amharic text cleaning and normalization ✅ Dataset quality validation ✅ Duplicate detection and removal ✅ Tokenizer fertility analysis ✅ Automatic train/validation/test splitting ✅ ChatML formatting ✅ QLoRA fine-tuning ✅ Training monitoring ✅ Automated evaluation ✅ GGUF export ✅ Hugging Face deployment ✅ Ollama deployment --- # Who Is This For? This notebook is useful for anyone who wants to fine-tune a model on Amharic data, including: * Agriculture chatbots * Healthcare assistants * Educational tutors * Customer support agents * Domain-specific AI assistants * General Amharic instruction models The notebook is domain-independent. Simply replace the dataset with your own Amharic data. --- # 🛠️ Technologies Used * Python * Unsloth * Transformers * TRL * PEFT * BitsAndBytes * Hugging Face * Datasets * Pandas * Google Colab --- # 📊 Dataset Requirements The notebook expects instruction-style data. Example: ```json { "user": "የስንዴ ተባይ እንዴት ማስቆጣጠር እችላለሁ?", "assistant": "የስንዴ ተባይን ለመቆጣጠር..." } ``` Minimum recommended size: * 5,000+ examples Good quality: * 20,000+ examples Excellent quality: * 50,000+ examples --- # Complete Workflow ## Step 1: Prepare Your Dataset Collect high-quality Amharic question-answer pairs. Examples: * User → Assistant conversations * FAQ datasets * Expert responses * Knowledge-base content --- ## Step 2: Upload Data Upload your dataset to: * Google Sheets * CSV * JSON The notebook loads the data automat …