Urz1/Amharic_Image_Captioning

# Amharic Image Captioning System A multimodal deep learning system that generates **Amharic language captions** for images using a Vision Transformer (ViT) encoder and GPT-2 decoder architecture, trained on the Flickr8k dataset with machine-translated Amharic captions. ## 🎯 Project Overview **First-ever image captioning system for Amharic language** - **Input:** Any image → **Output:** Grammatically correct Amharic description - **Architecture:** ViT encoder + GPT-2 decoder with cross-attention - **Dataset:** 40,460 Amharic captions for 8,091 Flickr8k images - **Performance:** 70% good-to-excellent caption quality ## 🚀 Quick Start ```bash # Setup python -m venv .venv source .venv/bin/activate # Linux/Mac pip install -r requirements.txt # Generate caption python demo.py path/to/your/image.jpg ``` **Example Output:** ``` Amharic Caption: አንዲት ትንሽ ልጅ በሮዝ ልብስ ደረጃዎችን እየወጣች ነው። English Translation: "A little girl in pink clothes is climbing stairs." ``` ## 📊 Results **Performance Metrics:** - **BLEU-4:** 0.089 (first baseline for Amharic) - **Training Time:** ~45 minutes on free Kaggle GPU - **Model Size:** 915MB checkpoint - **Inference:** 2-3 seconds per image on CPU **Quality Distribution:** - 25% Excellent (perfect descriptions) - 45% Good (minor inaccuracies) - 25% Acceptable (general understanding) - 5% Poor (significant errors) ## 🔬 Technical Details ### Architecture - **Vision Encoder:** ViT-Base-Patch16-224 (frozen, 86M params) - **Language Decoder:** GPT-2 with cross-attention (124M params) - **Bridge:** Linear projection layer (768→768 dimensions) ### Key Innovation Cross-attention mechanism allows GPT-2 to "see" image patches while generating Amharic text, despite GPT-2 having no native Amharic knowledge. ## 📚 Documentation - **DOCUMENTATION.md** - Complete technical documentation - **PROJECT_PRESENTATION.md** - Project overview slides - **STUDY_NOTES.md** - Learning resources for encoder-decoder architectures - **TRAINING_RESULTS.md** - Final …