NaomiMeseret/Amharic-LLM-finetuning-pipeline

# Amharic LLM Fine-Tuning Fine-tuning a Large Language Model for Amharic — a low-resource Semitic language spoken by 37 million people in Ethiopia — using QLoRA on LLaMA 3.2 1B. ## Overview Amharic is severely underrepresented in most large language model training corpora, accounting for less than 0.01% of CommonCrawl data. This project demonstrates a practical, low-budget approach to adapting an open-source LLM for Amharic English-to-Amharic translation using parameter- efficient fine-tuning. ## Key Findings - LLaMA's default tokenizer fragments Amharic words into an average of 11 raw byte tokens vs 2 tokens for equivalent English words — a 6.5x inflation ratio - QLoRA reduces VRAM requirement from ~56GB (full fine-tuning) to under 4GB for a 1B parameter model - 61,064 clean parallel Amharic-English pairs were prepared from the OPUS Bible corpus after preprocessing - Only 0.27% of model parameters are trained using LoRA adapters (3.4M out of 1.2B) ## Model - Base model: meta-llama/Llama-3.2-1B - Method: QLoRA (4-bit NF4 quantization + LoRA rank 16) - Task: English to Amharic translation - Hardware: Google Colab T4 GPU (free tier) ## Dataset Primary: OPUS Bible Corpus (61,084 parallel pairs) object.pouta.csc.fi Additional sources evaluated: - CC-100 Amharic: Cc100 - FLORES-200: flores200 - MasakhaNER: masakhaner2.0 ## Quick Start # 1. Clone the repository git clone github.com cd amharic-llm-finetune # 2. Install dependencies pip install -r requirements.txt # 3. Download and preprocess data python scripts/preprocess.py # 4. Train the model python scripts/train.py # 5. Run inference python scripts/inference.py ## Training Configuration | Parameter | Value | |-----------|-------| | Base Model | LLaMA 3.2 1B | | LoRA Rank | 16 | | LoRA Alpha | 32 | | Quantization | 4-bit NF4 | | Batch S …