Nuna7/Efficient-transfer-of-VLM-capabilities-in-low-resource-setting-through-mechanistic-interpretability
Domain:
natural language processingCreator:
Nun
Host:
# Efficient transfer and evaluation of vision language capabilities of low-resource languages through mechanistic interpretability of multilingual vision language model
---
## Project Structure
```
vlm_thesis/
├── configs/
│ └── settings.py # All global constants, paths, model IDs, task/lang defs
│
├── data/
│ ├── __init__.py
│ ├── base.py # Shared data utilities (image loading, splits)
│ ├── xm3600.py # XM3600 captioning loader
│ ├── xgqa.py # xGQA VQA loader
│ ├── marvl.py # MaRVL binary reasoning loader
│ └── pope_ml.py # Multilingual POPE loader
│
├── models/
│ ├── __init__.py
│ ├── loader.py # Model loading (Pangea + Qwen2.5-VL)
│ ├── inputs.py # Input preparation for both models
│ └── delta.py # Delta save / load / apply utilities
│
├── probes/
│ ├── __init__.py
│ ├── lang_layer_probe.py # RQ1: LIP + CLRA + FALD experiments
│ ├── mech_probes.py # RQ1: LJN + IDP-T proxy methods
│ └── binary_ldp.py # RQ1: Binary LDP proxy (MaRVL / POPE)
│
├── finetune/
│ ├── __init__.py
│ ├── trainer.py # Shared AdamW-8bit trainer (cosine LR, masking)
│ ├── single_layer.py # Single-layer FFN sweep (task_sweep)
│ ├── full_ft.py # Full model fine-tuning
│ └── combo_sweep.py # K-layer combination fine-tuning
│
├── evaluate/
│ ├── __init__.py
│ ├── metrics.py # CIDEr, exact-match, binary-acc, yes-ratio
│ ├── e1_preservation.py # E1: English capability (lmms-eval wrapper)
│ ├── e2_transfer.py # E2: Cross-task transfer specificity
│ └── e3_language.py # E3: Language specificity
│
├── scripts/
│ ├── run_lang_probe.py # Entry point: RQ1 mechanistic probing
│ ├── run_task_sweep.py # Entry point: single-layer FFN sweep
│ ├── run_combo_sweep.py # Entry point: K-layer combination sweep
│ ├── run_full_ft.py # Entry point: full model fine-tuning
│ …