A Leakage-Aware and Efficiency-Optimised Multimodal Fusion Selection Protocol for Low-Resource Image-Grounded Neural Machine Translation

The growing interest in multimodal neural machine translation (NMT) can be attributed to its ability to combine textual and visual information when source sentences are ambiguous, visually grounded, or context-dependent. This capability is particularly relevant to low-resource languages, where translation performance is constrained by scarce parallel data, morphological complexity, orthographic variation, and weak contextual grounding. Under these conditions, text-only NMT systems may struggle when sentences are short, ambiguous, or dependent on visual context. This study presents a leakage-aware and efficiency-optimised multimodal fusion selection protocol for English–Yoruba and English–Hausa image-grounded NMT by comparing five text-image fusion strategies under a common Transformer backbone. Whole-image features from Contrastive Language–Image Pre-training (CLIP ViT-B/16) and object-level features from Visual features in Vision-Language models with Faster Region-based Convolutional Neural Network detection (VinVL/Faster R-CNN) are integrated with textual representations. Non-dominated Sorting Genetic Algorithm II (NSGA-II) is used for multi-objective selection based on Bilingual Evaluation Understudy (BLEU), Character n-gram F-score with β = 2 (ChrF2), cached-feature inference time, and peak graphics processing unit (GPU) memory. The selected gated multimodal model achieves BLEU scores of 48.82 for English–Yoruba and 47.88 for English–Hausa, improving average BLEU from 45.85 to 48.35. Although hierarchical cross-modal attention and graph-based fusion achieve higher raw BLEU scores, they require greater inference time and memory. The selected gated model therefore provides a practical quality–efficiency trade-off, supported by human evaluation, ablation, and perturbation evidence showing that the gains are linked to meaningful image–sentence correspondence rather than parameter count alone.