Impact of Pre-trained Multilingual Language Models on Zero-shot Cross-lingual NER Transfer Performance

Multi-lingual language models (LM), such as mBERT, XLM-R, mT5, mBART, have been remarkably successful in enabling natural language tasks in low-resource languages through cross-lingual transfer from high-resource ones. In this work, we try to better understand how such models, specifically mT5, transfer *any* linguistic and semantic knowledge across languages, even though no explicit cross-lingual signals are provided during pre-training. Rather, only unannotated texts from each language are presented to the model separately and independently of one another, and the model appears to implicitly Research goal: To what extent does the choice of pre-trained multilingual language model (e.g., mBERT, XLM-R, Bloom) affect the zero-shot cross-lingual transfer performance of teacher-student NER models on the CoNLL-2003 benchmark when evaluated on zero-resource languages? Autonomous synthesis report generated by Assignee Research. Tribunal consensus score: 8.3/10. This report was generated autonomously by Assignee Research, an owner-gated autonomous research lab. The content synthesizes findings from peer-reviewed papers. Tribunal score: 8.3/10.