surrey-nlp/ALOPE-RL

A policy-based RL framework for Quality Estimation of Machine Translation on low-resource languages. # ALOPE-RL ALOPE-RL is a policy-based reinforcement learning framework for **Machine Translation Quality Estimation (QE)**. It leverages the **GRPO (Group Relative Policy Optimization)** algorithm to train efficient adapters for Large Language Models (LLMs), enabling them to generate precise quality scores, error categorizations, and detailed **Translation Quality Remarks (TQR)**. The framework is specifically designed to address gaps in low-resource language evaluation (e.g., English -> Malayalam, English -> Hindi) by utilizing rewards derived from Direct Assessment (DA) scores and contextual annotator comments. --- ## 🚀 Key Features - **Policy-Based Reinforcement Learning**: Implements the **GRPO** algorithm via the `trl` library, enabling high-performance policy optimization without the overhead of a separate critic model. - **TQR-Augmented Training**: Leverages **Translation Quality Remarks (TQR)** -- contextual annotator comments to drive better judgment and explainability in QE outputs. - **Multi-Component Reward System**: Models are optimized using a weighted reward aggregation system: - **DA Score Accuracy**: Proximity to ground-truth Direct Assessment (DA) scores (Exact Score & Score Bin). - **Error Categorization**: Accuracy in identifying specific error types (e.g., Mistranslation, Addition, Untranslated). - **Description Quality**: Semantic similarity of generated TQR via **BERTScore**. - **Formatting & Length**: Ensures output adheres to a strict XML structure and maintains optimal verbosity. - **Efficient Fine-Tuning**: Built on **Unsloth**, utilizing 4-bit quantization and LoRA adapters to achieve state-of-the-art results with compact LLMs (≤4B parameters). --- ## 📂 Repository Structure | File | Description | | :--- | :--- | | `rlqe_word_tags.py` | Training script utilizing word-level quality tags (OK/BAD) for English-Hindi evaluation. | | `rlqe_weak_annotations.py` | Training script utilizing Translation Quality Remarks (TQR) for English-M …