Ensembling of Distilled Models from Multi-task Teachers for Constrained Resource Language Pairs
Domaine:
natural language processingType de record:
papermodel
Créateur:
HenGadAbdElM
Hôte:
This paper describes our submission to the constrained track of WMT21 shared news translation task. We focus on the three relatively low resource language pairs Bengali to and from Hindi, English to and from Hausa, and Xhosa to and from Zulu. To overcome the limitation of relatively low parallel data we train a multilingual model using a multitask objective employing both parallel and monolingual data. In addition, we augment the data using back translation. We also train a bilingual model incorporating back translation and knowledge distillation then combine the two models using sequence-to-sequence mapping. We see around 70% relative gain in BLEU point for English to and from Hausa, and around 25% relative improvements for both Bengali to and from Hindi, and Xhosa to and from Zulu compared to bilingual baselines.
Similaires
Efficient Test Time Adapter Ensembling for Low-resource Language VarietiesMulti-source Intermediate-task Training for Low-resource XTREME Language GeneralizationA Multi-Task Benchmark for Abusive Language Detection in Low-Resource SettingsA multi-task learning framework for sentiment analysis and news classification for low-resource languageDeep Conditional Census-Constrained Clustering (DeepC4) for Large-scale Multi-task Disaggregation of Urban MorphologyImpact of Intermediate-Task Training on Low-Resource Languages in Multilingual Language Models