Beyond parallel data: decipherment for better quality machine translation

Thanks to the use of parallel data and advance machine learning techniques, we have seen tremendous improvement in the field of machine translation over the past 20 years. However, due to lack of sufficient parallel data, the quality of machine translation is still far from satisfying for many language pairs and domains. In general, it is easier to obtain non-parallel data, and much work has tried to discover word level translations from non-parallel data. Nonetheless, improvements to machine translation have been limited. In this work, I follow a decipherment approach to learn translations from non-parallel data and achieve significant gains in the quality of machine translation. ? First of all, I apply slice sampling to Bayesian decipherment to make it highly scalable and accurate, making it possible to decipher billions of tokens with hundreds of thousands of word types at high accuracy. Then, when it comes to deciphering foreign languages, I introduce dependency relations to address the problems of word reordering, insertion, and deletion. Experiments show that dependency relations help improve Spanish/English deciphering accuracy by over 5-fold. Last but not least, this accuracy is further doubled when word embeddings are used to incorporate more contextual information. ? With faster and more accurate decipherment algorithms, I decipher large amounts of monolingual data to improve the state-of-the-art machine translation systems in the scenario of domain adaptation and low density languages. Through experiments, I show that decipherment finds high quality translations for out-of-vocabulary words in the task of domain adaptation, and help improve word alignment when the amount of parallel data is limited. I observe up to 3.8 point and 1.9 point Bleu gain in Spanish/French and Malagasy/English machine translation experiments respectively. ? In the end, I release a decipherment package?MonoGIZA, which finds word level translations from monolingual corpora. It serves the purpose to facilitate future research in replicating and advancing the work described in this thesis.