shuokabe/PaSeMiLL

Parallel Sentence Mining for Low-Resource Languages # PaSeMiLL: Parallel Sentence Mining for Low-Resource Languages This repository contains the published artefacts for parallel sentence mining from (Okabe and Fraser, 2025) and (Okabe et al., 2025). ## Data section The `data` folder contains the raw (unprocessed) datasets and the BUCC-style files for Upper and Lower Sorbian for (Okabe and Fraser, 2025). The dataset from (Okabe et al., 2025) is available on its dedicated repository: Belopsem. ## Code section The `code` folder contains useful code to use with the original UnsupPSE pipeline (Hangya and Fraser, 2019). The pre-training of XLM-R with Upper Sorbian data is addressed in the `pretraining` subfolder. ## How to use? The full pipeline is available in the `code/mine_bucc_full_xlmr.sh` file. Once your two monolingual corpora are ready (one sentence per line): 1. Convert the sentences into embeddings using the backend language model of your choice (e.g., XLM-R, Glot500, or pre-trained). 2. Compute similarity scores between your source and target sentences. 3. Filter the output sentence pairs based on a defined threshold (hyperparameter). ## Licence The repository has the MIT licence (e.g., `code` folder), *except* for the benchmark dataset. The dataset (`data` folder) is released with a CC BY-NC-SA licence (data from (Okabe and Fraser, 2025)). ## Citations For the updated version of the pipeline, please use the following citation (from the ACL Anthology): ``` @inproceedings{okabe-etal-2025-improving, title = "Improving Parallel Sentence Mining for Low-Resource and Endangered Languages", author = {Okabe, Shu and H{\"a}mmerl, Katharina and Fraser, Alexander}, editor = "Che, Wanxiang and Nabende, Joyce and Shutova, Ekaterina and Pilehvar, Mohammad Taher", booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)", month = jul, year = "2025", address = "Vienna, Austria", publisher = "Association for Computational Linguistics", url = "htt …