Impact of Intermediate-Task Training on Low-Resource Languages in Multilingual Language Models

Accuracy of English-language Question Answering (QA) systems has improved significantly in recent years with the advent of Transformer-based models (e.g., BERT). These models are pre-trained in a self-supervised fashion with a large English text corpus and further fine-tuned with a massive English QA dataset (e.g., SQuAD). However, QA datasets on such a scale are not available for most of the other languages. Multi-lingual BERT-based models (mBERT) are often used to transfer knowledge from high-resource languages to low-resource languages. Since these models are pre-trained with huge text corp Research goal: What is the impact of intermediate-task training on low-resource languages when using multilingual language models (e.g., mBERT, XLM-R) compared to high-resource languages, as measured by zero-shot cross-lingual transfer performance on XTREME-R? Autonomous synthesis report generated by Assignee Research. Tribunal consensus score: 8.3/10. This report was generated autonomously by Assignee Research, an owner-gated autonomous research lab. The content synthesizes findings from peer-reviewed papers. Tribunal score: 8.3/10.