farunawebservices/igala-mbert-interpretability

Mechanistic interpretability analysis of mBERT attention patterns in low-resource Igala-English translation # 🔬 Mechanistic Interpretability Analysis Getting into mBERT's internal mechanisms during Igala-English translation. Visualizes attention patterns, analyzes token alignments, and explores how transformers handle morphologically complex low-resource languages. ## 🎯 Overview This project investigates **how transformer models actually work** when translating between English and Igala (a low-resource Nigerian language). By visualizing attention heads and analyzing internal representations, we gain insights into: - How the model aligns tokens across languages - Which attention heads focus on syntactic vs semantic features - How morphological complexity affects attention patterns - Where the model struggles with low-resource data ## 🚀 Live Demo Explore attention patterns: huggingface.co ## 📊 Features - ✅ Layer-by-layer attention visualization (12 layers × 12 heads = 144 attention matrices) - ✅ Token-level alignment heatmaps - ✅ Interactive Plotly visualizations - ✅ Comparative analysis across language pairs - ✅ Morphological feature tracking ## 🛠️ Tech Stack - **Model**: `bert-base-multilingual-cased` (mBERT) - **Framework**: PyTorch, TransformerLens - **Visualization**: Plotly, Matplotlib - **Frontend**: Streamlit - **Analysis**: NumPy, Pandas ## 📦 Installation ```bash # Clone the repository git clone github.com cd igala-mbert-interpretability # Install dependencies pip install -r requirements.txt # Run the app streamlit run app.py 🔍 Usage from interpretability import AttentionAnalyzer # Initialize analyzer analyzer = AttentionAnalyzer(model_name="bert-base-multilingual-cased") # Analyze attention patterns attention_map = analyzer.get_attention( source_text="Ọma ẹdu la", # Igala: "Good morning" target_text="Good morning", layer=6, head=3 ) # Visualize analyzer.plot_attention_heatmap(attention_map) 📈 Key Findings Attention Pattern Observation …