Nuna7/Efficient-transfer-of-VLM-capabilities-in-low-resource-setting-through-mechanistic-interpretability

Domaine:

natural language processing

Créateur:

Nun

Hôte:

# Efficient transfer and evaluation of vision language capabilities of low-resource languages through mechanistic interpretability of multilingual vision language model --- ## Project Structure ``` vlm_thesis/ ├── configs/ │ └── settings.py # All global constants, paths, model IDs, task/lang defs │ ├── data/ │ ├── __init__.py │ ├── base.py # Shared data utilities (image loading, splits) │ ├── xm3600.py # XM3600 captioning loader │ ├── xgqa.py # xGQA VQA loader │ ├── marvl.py # MaRVL binary reasoning loader │ └── pope_ml.py # Multilingual POPE loader │ ├── models/ │ ├── __init__.py │ ├── loader.py # Model loading (Pangea + Qwen2.5-VL) │ ├── inputs.py # Input preparation for both models │ └── delta.py # Delta save / load / apply utilities │ ├── probes/ │ ├── __init__.py │ ├── lang_layer_probe.py # RQ1: LIP + CLRA + FALD experiments │ ├── mech_probes.py # RQ1: LJN + IDP-T proxy methods │ └── binary_ldp.py # RQ1: Binary LDP proxy (MaRVL / POPE) │ ├── finetune/ │ ├── __init__.py │ ├── trainer.py # Shared AdamW-8bit trainer (cosine LR, masking) │ ├── single_layer.py # Single-layer FFN sweep (task_sweep) │ ├── full_ft.py # Full model fine-tuning │ └── combo_sweep.py # K-layer combination fine-tuning │ ├── evaluate/ │ ├── __init__.py │ ├── metrics.py # CIDEr, exact-match, binary-acc, yes-ratio │ ├── e1_preservation.py # E1: English capability (lmms-eval wrapper) │ ├── e2_transfer.py # E2: Cross-task transfer specificity │ └── e3_language.py # E3: Language specificity │ ├── scripts/ │ ├── run_lang_probe.py # Entry point: RQ1 mechanistic probing │ ├── run_task_sweep.py # Entry point: single-layer FFN sweep │ ├── run_combo_sweep.py # Entry point: K-layer combination sweep │ ├── run_full_ft.py # Entry point: full model fine-tuning │ …

Visit

github.com

Similaires

Impact of Sequential Fine-Tuning on mT5 Reasoning Capabilities in Low-Resource Language TransferEfficient Model Development through Fine-tuning TransferVLURes: Benchmarking VLM Visual and Linguistic Understanding in Low-Resource LanguagesContrastive Alignment of Vision to Language Through Parameter-Efficient Transfer LearningPredicting stillbirth in a low resource settingpaul-rottger/efficient-low-resource-hate-detection

Impact of Sequential Fine-Tuning on mT5 Reasoning Capabilities in Low-Resource Language Transfer

Euphemisms are culturally variable and often ambiguous, posing challenges for language models, espec

Efficient Model Development through Fine-tuning Transfer

Modern LLMs struggle with efficient updates, as each new pretrained model version requires repeating

VLURes: Benchmarking VLM Visual and Linguistic Understanding in Low-Resource Languages

Vision Language Models (VLMs) are pivotal for advancing perception in intelligent agents. Yet, evalu

Contrastive Alignment of Vision to Language Through Parameter-Efficient Transfer Learning

Contrastive vision-language models (e.g. CLIP) are typically created by updating all the parameters

Predicting stillbirth in a low resource setting

Abstract Background Stillbirth is a major contributor to perinatal mortality and it is particularly

paul-rottger/efficient-low-resource-hate-detection

Röttger et al. (EMNLP 2022): "Data-Efficient Strategies for Expanding Hate Speech Detection into Und