YingWANGG/xMDETR

Adapting Grounded Visual Question Answering Models to Low Resource Languages # Adapting Grounded Visual Question Answering Models to Low Resource Languages We propose xMDETR, a multi-lingual grounded vision-language model based on the state-of-the-art model MDETR, by adapting it to new languages without machine-translated data, while also keeping most of the pre-trained weights frozen (only the embedding layer and adapters in the text encoders are updated). There are two or three streams of data in the proposed cross-lingual transfer. The first stream only includes textual data in the target language and is fed to the text encoder to compute the MLM loss. The second stream consists of images from GQA and corresponding code-switched questions, fed into pre-trained MDETR for QA loss. For languages with existing image-caption datasets (such as German and Chinese), we have an additional data stream to compute the contrastive loss. For more details, please see the paper: Adapting Grounded Visual Question Answering Models to Low Resource Languages by Ying Wang, Jonas Pfeiffer, Nicolas Carion, Yann LeCun, Aishwarya Kamath. ### Checkpoints We provided checkpoints trained with MLM and code-switch QA (and contrastive loss if applicable). The test accuracy is reported as zero-shot results in the paper. ### Data Preparation 1. Update the path of the GQA dataset in gqa.json. "vg_img_path" should point to the directory where GQA images are stored. The GQA annotation files (to obtain these files, see MDETR) should be stored in the folder ```annotations```. 2. Obtain annotation files from xGQA. Store them under ```annotations/fewshot```. ### Training (MLM + code-switch QA) To facilitate training and evaluation, we provided a copy of a subset of MDETR code in this repo. The code for the model and dataloader has been modified for cross-lingual transfer implemented in this repo. 1. We have provided bilingual dictionaries for each language from xGQA in ```data/fasttext/```. The word-level translations are based on bilingual dictionaries from MUSE, and t …