thompsonb/fairseq-smrt

Code for "Simulated Multiple Reference Training Improves Low-Resource Machine Translation" # Simulated Multiple Reference Training (SMRT) This repo contains a fork of fairseq sufficient to replicate the experiments in Simulated Multiple Reference Training Improves Low-Resource Machine Translation by Huda Khayrallah, Brian Thompson, Matt Post, Philipp Koehn. Here's the abstract from the paper: >Many valid translations exist for a given sentence, yet machine translation (MT) >is trained with a single reference translation, exacerbating data sparsity in low-resource settings. >We introduce Simulated Multiple Reference Training (SMRT), >a novel MT training method that approximates the full space of possible translations >by sampling a paraphrase of the reference sentence from a paraphraser >and training the MT model to predict the paraphraser’s distribution over possible tokens. >We demonstrate the effectiveness of SMRT in low-resource settings when translating to English, >with improvements of 1.2 to 7.0 BLEU. We also find SMRT is complementary to back-translation. An illustration of our method is below. Each time a target sentence in the training data is used, a paraphrase(blue) is sampled from the many possible paraphrases (grey). The training objective also takes into account the distribution over all the possible options (green) along the sampled path. Our method is implemented as a fairseq criterion, enabled via: ``` --criterion smrt_cross_entropy ``` It takes in a pretrained Fairseq paraphraser and data directory for the corresponding dictionary: ``` --paraphraser-model /path/to/paraphraser/paraphraser.pt --paraphraser-data-dir /directory/containing/directory/ ``` During training, our new objective is mixed with standard label smoothed cross entropy training. To control the fraction of the time the new objective is used: ``` --prob-use-smrt 0.5 ``` Finally, to control the amount of variation introduced when sampling a path from the paraphraser: ``` --paraphraser-sample-topN 100 ``` # Installation Instructions To install in a conda environm …