Small Batch Sizes Improve Training of Low-Resource Neural MT
Domain:
natural language processingRecord type:
paper
Creator:
AtrPop
Host:
We study the role of an essential hyper-parameter that governs the training of Transformers for neural machine translation in a low-resource setting: the batch size. Using theoretical insights and experimental evidence, we argue against the widespread belief that batch size should be set as large as allowed by the memory of the GPUs. We show that in a low-resource setting, a smaller batch size leads to higher scores in a shorter training time, and argue that this is due to better regularization of the gradients during training. To be published in 18th International Conference on Natural Language Processing (ICON 2021)
Similar
Hybrid Batch Training for Zero-Shot Retrieval in Low-Resource Languagestyracs18/low-resource-mtHybrid Batch Training for Zero-Shot Retrieval in Low-Resource MIRACL LanguagesHybrid Batch Training for Zero-Shot Low-Resource Language Retrieval in XTREMEHybrid Batch Training and Zero-Shot Retrieval Accuracy in Low-Resource XQuADHybrid Batch Training for Zero-Shot Low-Resource Language Retrieval on XLSD