Robustness of Zero-Shot Cross-Lingual Performance in Low-Resource Languages and Typological Distance Correlation

Intermediate-task training---fine-tuning a pretrained model on an intermediate task before fine-tuning again on the target task---often improves model performance substantially on language understanding tasks in monolingual English settings. We investigate whether English intermediate-task training is still helpful on non-English target tasks. Using nine intermediate language-understanding tasks, we evaluate intermediate-task transfer in a zero-shot cross-lingual setting on the XTREME benchmark. We see large improvements from intermediate training on the BUCC and Tatoeba sentence retrieval tas Research goal: How robust are zero-shot cross-lingual performance gains from English intermediate-task training when applied to low-resource languages in XTREME-R, and does this robustness correlate with the linguistic typological distance between English and the target language (e.g., using metrics like LASER distance)? Autonomous synthesis report generated by Assignee Research. Tribunal consensus score: 9.3/10. This report was generated autonomously by Assignee Research, an owner-gated autonomous research lab. The content synthesizes findings from peer-reviewed papers. Tribunal score: 9.3/10.