Unsupervised Language Model Adaptation for Low-Resource Languages

This paper introduces a two-way neural machine translation system from Bengali to English and vice versa, with two different models: a Transformer model coded entirely in PyTorch, and a pre-trained T5ForConditionalGeneration model from Hugging Face. The system is designed to tackle the issues of low-resource language processing and can translate in both directions — Bengali→English and English→Bengali. The special Transformer model adopts the same encoder-decoder architecture with multi-head self-attention and positional encoding, but trained on a hand-curated Bengali-English parallel corpus. For Bengali, a rule-based tokenizer is applied, and English is tokenized using SpaCy. Concurrently, we compare the T5 model fine-tuned on the same data as a baseline for pretrained transformer performance. Both models are measured in terms of BLEU, METEOR, and TER scores. The findings indicate that the from-scratch model attains competitive translation performance, whereas the pretrained model shows improved convergence and generalization, which makes this a comparative study of pretrained and custom methods for bilingual translation in low-resource environment.