ruoyuxie/noisy_parallel_data_alignment

Enhanced awesome-align for low-resource languages and noise simulation: arxiv.org # Enhanced awesome-align for low-resource languages ## About this repository `Enhanced awesome-align` is a word alignment tool built upon on awesome-align. It is specifically designed for noisy ***low-resource languages***. This repo contains the instruction and code for noise simulation and extracting the alignment. We also release the ***gold*** word alignment pairs between an endangered language, Griko, and translations in Italian. The gold alignment data can be found on `alignment_data/griko-italian` 👉 See our paper Noisy Parallel Data Alignment for more details 👈 ### Installation Install the requirements by running the following command: ```bash pip install -r requirements.txt python3 setup.py install ``` ## Noise simulation To simulate text noise, we need to *first* calculate the noise probability and *then* generate synthetic data. First, run the following command to calculate noise probability: ```bash python3 noise_simulation/noise_analysis.py ``` `parallel_data` is the clean and noisy version of the same text that is split by ` ||| `. Example can be found on `noise_simulation/example/griko_clean_noisy.txt` `noisy_probability_data` is the produced noise probability file that will be used for the next step to create synthetic data. Next, to generate synthetic noisy data, run the following command: ```bash python3 noise_simulation/noise_adding.py ``` `clean_input_file` is the clean text that will be adding text noise. Example can be found on `noise_simulation/example/griko_clean.txt`. `output_file` is the output file that will contain the synthetic data. `noisy_probability_data` is the noise probability file from the previous step. ## Running enhanced awesome-align #### Input format Similar to original `awesome-align`, the inputs data should be *tokenized* parallel data. The source and target sentences are separated by ` ||| `. Example data can be found on `alignment_data/griko-italian/all.txt`. #### Extracting alignments To extract alignment, p …