ArmelRandy/topxgen

[EMNLP 2025 Findings] TopXGen: Topic-Diverse Parallel Data Generation for Low-Resource Machine Translation # Official Repo of TopXGen Official implementation of TopXGen: Topic-Diverse Parallel Data Generation for Low-Resource Machine Translation with code, prompts and model outputs. # Table of Contents 1. Overview of TopXGen 2. Installation 3. Experiments - 3.1 Dataset Generation - 3.2 Decontamination - 3.3 Dataset Description - 3.4 Training - 3.5 Evaluation - 3.6 Ablation studies 4. Contributions 5. Aknowledgements 6. Citations # Overview We propose to generate a parallel datasets by leveraging language models. Given a target language, we can prompt a generator **G** to get generate paragraphs on a range of topics. The obtained paragraphs are then splitted into sentences and cleaned to obtain a sentence-level monolingual dataset. A back-translator **BT** is then used to [back-]translate those sentences into a source language of interest (typically a high-resource language such as English). Given the generated dataset, which we term as *TopXGen* dataset, you can train a small LM (e.g. LLaMA 2 7B, LLaMA 3 8B, GEMMA 3 4B PT) to perform MT between the source language and the languages generated. You can also used it as a selection pool for In-Context Learning. # Installation This repository relies on vLLM and Transformers. It also supports Cohere and OpenAI models via their respective API. Before using this repository, make sure to install PyTorch in accordance to the characteristics of your device. The rest of the required libraries can be installed via ``` git clone github.com cd topxgen/ pip install -r requirements.txt ``` In case you want to use SONAR and/or BLASER, you will need to do `pip install sonar-space`. Feel free to refer to SONAR's repository for more information. You might also require FlashInfer (`pip install flashinfer==0.1.2 -i flashinfer.ai`) if you work with Gemma models and Flash Attention (`MAX_JOBS=4 pip install flash-attn --no-build-isolation --no-cache-dir`) for fast inference. Rece …