Machine Translation with Neural Networks Based on a Transformer

Over the same period, thanks to deep neural networks, particularly in machine translation tasks that fall under natural language process (NLP), there have been major advances. Context is not something that lies well with traditional systems based on statistical models or tranlation memory and they also struggle with the level of fluency in translation. Through the invention of their Transformer models, Vaswani et al. This has also made great strides in improving both precision and performance [28] from 2017. In this paper, we explore how to adapt pre-trained Transformer models into low-resource languages. For lower-engaged languages, we want to introduce hybrid methods in machine translation systems using multi-task learning and also transfer-learning with different metrics. To address this, we exploit the high-resource languages for training and introduce representation reduction techniques to be able to better handle some low-resourced cases. Our results show that multi-task training improves BLEU scores by a large margin, especially for more scarce languages such as Swahili and Amharic. Our results exemplify the power of combining effective multi-task training with transfer learning in low-resource language translation.