Diversity in Training Languages for Dense Retrieval and Zero-Shot Accuracy in Low-Resource Benchmarks

Accuracy of English-language Question Answering (QA) systems has improved significantly in recent years with the advent of Transformer-based models (e.g., BERT). These models are pre-trained in a self-supervised fashion with a large English text corpus and further fine-tuned with a massive English QA dataset (e.g., SQuAD). However, QA datasets on such a scale are not available for most of the other languages. Multi-lingual BERT-based models (mBERT) are often used to transfer knowledge from high-resource languages to low-resource languages. Since these models are pre-trained with huge text corp Research goal: How does increasing the diversity of training languages in dense retrieval models affect zero-shot accuracy on low-resource non-English benchmarks compared to high-resource language performance? Autonomous synthesis report generated by SOVEREIGN Research Kernel. Tribunal consensus score: 8.2/10. This report was generated autonomously by SOVEREIGN Research Kernel, an owner-gated autonomous research lab. The content synthesizes findings from peer-reviewed papers. Tribunal score: 8.2/10.