mcgwilmo/Leveraging-Portuguese-Latent-Spaces-for-Kriolu-to-English-Translation-with-Monolingual-LM-Priors

Experiments in low-resource Kriolu→English neural MT. We build on “Language Model Prior for Low-Resource NMT”, using Portuguese-initialized BERT encoders and English LM priors, and compare against from-scratch baselines on the KreyolMT dataset. This project investigates low-resource translation for Cape Verdean Kriolu by leveraging its historical and structural proximity to Portuguese and improving English generation through a language model prior. We propose an encoder–decoder framework that uses Portuguese-based transformer encoders as a shared latent space and frozen English language models as decoder priors. We evaluate three Portuguese encoders (BERTimbau, Albertina PT-PT, and Albertina PT-BR) paired with three English models (BERT-base, RoBERTa-base, and Electra), yielding nine encoder–decoder configurations. Models are trained with AdamW using cross-entropy loss augmented by a KL-divergence regularizer that aligns translation model outputs with softened English language model distributions. Performance is assessed using BLEU and chrF on validation and test splits, alongside a baseline employing a randomly initialized latent-space encoder. Results show improvements in validation loss and test BLEU and chrF scores across all pretrained Portuguese encoder pairings relative to the baseline, with Portuguese BERT-based encoders delivering the most consistent BLEU gains and Brazilian Portuguese latent spaces exhibiting particularly strong chrF performance. These findings suggest that related-language latent representations, combined with an English language model prior, can meaningfully improve translation quality and semantic stability in low-resource settings.