lerato1ofwan/760-lost-in-translation

This project evaluates translation quality improvements in low-resource African languages (*Sepedi*, Hausa, Xitsonga, isiZulu) using corrected FLORES datasets. We assess NLLB model outputs with state-of-the-art metrics (BLEU, chrF++, COMET, BLEURT) and apply explainability techniques (SHAP, LIME) to analyze translation errors and dataset integrity. 🌍 Machine Translation Evaluation Using the Corrected FLORES Datasets Group 19 – COS760 UP Natural Language Processing 2025 🏃 How to run: -Clone the github repository. -Execute "Lost_In_Translation_Notebook.ipynb" notebook file: Upload and run all cells either in Google Colab T4 runtime, or locally in Jupyter Notebook or equivalent. -We strongly advice using Google Colab's T4 runtime if local Jupyter Notebook is not powered by a GPU - the NLLB 3B model requires a GPU for faster processing. 📌 Project Overview This project evaluates the quality of machine translation (MT) systems before and after the correction of the FLORES dataset for four low-resource African languages: Hausa, Sepedi, Xitsonga, and isiZulu. Using both automated evaluation metrics and explainability techniques, we identify improvements in translation quality and diagnose common translation errors. The ultimate goal is to validate and strengthen dataset integrity for better MT outcomes. 🧠 Key Objectives -Assess translation performance before vs. after dataset corrections. -Apply and compare state-of-the-art evaluation metrics: BLEU, chrF++, COMET, BLEURT. -Use explainability methods (e.g. SHAP, error spans) to explore translation errors. -Investigate dataset validation strategies for low-resource language corpora. 🧪 Dataset -We use the Corrected FLORES Dataset provided by the Masakhane community: -Languages: Hausa, Sepedi, Xitsonga, isiZulu -Task: Evaluate translations generated by NLLB models using both the original and corrected test sets. 📊 Evaluation Metrics We use both automatic metrics and human-aligned scoring systems to assess quality: Metric Type Description : BLEU Lexical / N-gram Measures n-gram precision vs. reference translations. Common in MT benchmarking. chrF++ Character-level Captures word morphology better; more reliable for morphologically rich languages. COMET Neural / Semantic Learned metric aligned to human judgments. Considers source and hypothesis context. BLE …