eliplutchok/bemba-to-english-python

# Bemba to English Translation Evaluation Repository This repository contains the code and data used to evaluate and rank various models for translating Bemba to English. It automates the process of generating translations, computing evaluation metrics, and preparing data for human-like judgments, utilizing both traditional evaluation metrics and some novel techniques. **Website**: bemba-to-english.netlify.app **Frontend Repository**: Bemba to English React App --- ## Table of Contents - Overview - Repository Structure - Metrics and Methodologies - Judgment Battles - Consistency Battles - BERTScore Battles - Similarity Score Battles - Data Storage - Scripts and Workflow - Generating Translations - Adding Evaluation Metrics - Preparing Judgment Files - Main Script - Display and Analysis Tools - Prompts Used - Translation Prompt - Judgment Prompt - Consistency Judgment Prompt - How to Use - Additional Information --- ## Overview This repository automates the evaluation of translation models by generating translations, computing evaluation metrics, and preparing data for human-like judgments. The models are evaluated using the test set from the Big C dataset, and their rankings are determined using ELO ratings computed from various metrics. We introduce **Consistency Battles**, a novel method to evaluate the quality of translation models without relying on reference translations. By assessing the consistency of a model's outputs when generating translations under varying conditions, we can infer the reliability and accuracy of the model. This approach does not require any target/reference text, making it particularly useful when high-quality reference translations are scarce or unavailable. Our evaluation demonstrates that the ELO ratings derived from Consistency Battles closely align with those from traditional metrics such as BERTScore and human-like judgments, validating the effectiveness of our novel approach. The results of our evaluations are availabl …