chibeze01/MAD-X_Fusion

Multi adapter cross lingual transfer with Fusion training framework, demonstrating improved performance on low resource languages beating mBERT on low resource language on Squad style question answering task. Here is a possible readme for the PDF, based on the current web page context: # MAD-X Fusion: A Framework for Cross-Lingual Transfer in Low-Resource Languages This repository contains the code and data for the paper "Enhancing Cross-Lingual Transfer in Low-Resource Languages: Leveraging Parameter-Efficient Adapters for Multilingual Extractive Question Answering" by Chibeze Joseph Nwangwu. ## Abstract In this paper, we present MAD-X Fusion, a cross-lingual learning framework that leverages parameter-efficient adapters to improve performance on low-resource languages. We focus on the extractive question answering task, which requires language comprehension and contextual inference. Our framework consists of two steps: 1) training language-agnostic, task-specific adapters using the MAD-X framework, which combines language and invertible adapters, and 2) integrating AdapterFusion, a technique that allows for the composition of multiple adapters, to train a fusion layer on a source language dataset. We evaluate our framework on five languages (English, Hindi, Bengali, Swahili, and Icelandic) and show that it outperforms the baseline model (MAD-X) in both zero-shot and few-shot settings. We also analyze the effectiveness of different adapter compositions and provide insights into the benefits of AdapterFusion. ## Requirements - Python 3.6 or higher - PyTorch 1.8 or higher - Transformers 4.5 or higher - AdapterHub 2.1 or higher - Datasets 1.6 or higher ## Data We use the following datasets for our experiments: - SQuAD v1.1 (Rajpurkar et al., 2016) for English - iNLTK (Gupta et al., 2019) for Hindi - BengaliQA (Hasan et al., 2020) for Bengali - SwahiliQA (Ogueji et al., 2022) for Swahili - IceQuAD (Steingrímsson et al., 2020) for Icelandic The datasets can be downloaded from Hugging Face Datasets or from their respective sources. ## Models We use the following pre-trained multilingual models as our base models: - mBERT (Devlin et al., 2019) We use AdapterHub t …