itsShyPie/COS760-Group58-FLORES-MT-Evaluation

Re-evaluation of MT systems against the corrected FLORES dataset for four African languages # COS760 Group 58 - FLORES MT Re-Evaluation ## Overview Re-evaluating four machine translation systems against the **original** vs. **corrected** FLORES+ dataset for four African languages: **Hausa** (`hau_Latn`), **Northern Sotho** (`nso_Latn`), **Xitsonga** (`tso_Latn`), and **isiZulu** (`zul_Latn`). Each system translates the FLORES+ devtest set in both directions (English ↔ target language) using both the original and corrected references, and is scored with **BLEU**, **chrF**, and **COMET (Unbabel/wmt22-comet-da)**. The unique angle is the side-by-side comparison of scores under the original vs. corrected references to show how reference quality affects measured system performance. ## Team - Shahil Parbhoo Narsing - Matthew Gravette - Lesego Senamela ## Contents of the Zip File ``` README.md This file NLLB_200/ Meta NLLB-200 distilled 600M MADLAD_400/ Google MADLAD-400-3B-MT Helsinki/ Helsinki-NLP Opus-MT Google_Translate/ Google Translate API ``` Each system folder contains: - ` .ipynb` - the notebook that installs dependencies, downloads the data, runs translation, and computes scores. - `hau.csv`, `nso.csv`, `tso.csv`, `zul.csv` - per-language translation outputs with columns: `original`, `corrected`, `source_eng`, `original_english`, `corrected_english`, `translated_from_english`. - `bleuScores.csv`, `chrfScores.csv`, `COMETScores.csv` - aggregated scores across all four languages and both reference sets. ## Setup Instructions 1. Install **Python 3.10+** and **Jupyter** (or VS Code with the Jupyter extension). 2. (Recommended) Create a virtual environment: ``` python -m venv .venv .venv\Scripts\activate # Windows source .venv/bin/activate # Linux/macOS ``` 3. A CUDA-capable GPU is recommended for `NLLB_200`, `MADLAD_400`, and `Helsinki`. Notebooks were developed on an NVIDIA RTX 5070 (CUDA 12.8). 4. Each notebook installs its own depende …