janetzki/GUIDE

Create semantic domain dictionaries for low-resource languages # GUIDE: Creating Semantic Domain Dictionaries for Low-Resource Languages Code for GUIDE: Creating Semantic Domain Dictionaries for Low-Resource Languages, published at SIGTYP 2024. ## Our presentation (Click to open the YouTube video) (11:26 min) ## Overview Existing (black) semantic domain dictionary entries in FLEx and correct (green) and incorrect (red) new predictions: The upper image shows three entries that GUIDE added to the English dictionary and the lower image shows seven entries for the same semantic domain question in the newly created Mina-Gen dictionary. ## Requirements You need the Conda package manager to install the requirements. Furthermore, you need Git LFS to setup the repository. To install the requirements: ```setup chmod +x setup.sh ./setup.sh ``` This setup has been tested on an ASUS machine ESC8000 G4 with Ubuntu 22.04. ## Preprocessing Pipeline You can skip the preprocessing and directly start to train the model by using the prepared file `final_mag.cpickle`. (MAG stands for "Multilingual Alignment Graph".) If you want to reproduce the preprocessing, run: ```preprocess conda activate guide_env python -m src.preprocess --output-directory data/0_state/ && python -m src.gnn.refine_mag --input-mag-directory data/0_state/ --output-mag-file final_mag.cpickle ``` Note that the preprocessing does not include the Igbo and Gen-Mina languages because the source Bible translations are copyrighted. ## Training To train GUIDE, run this command: ```train conda activate guide_env CUDA_VISIBLE_DEVICES=0 python -m src.gnn.train --input-mag-file final_mag.cpickle --output-model-file my_trained_model.bin --output-data-split-file my_data_split.bin ``` ## Evaluation To evaluate GUIDE, run: ```eval conda activate guide_env CUDA_VISIBLE_DEVICES=0 python -m src.gnn.eval --input-mag-file final_mag.cpickle --input-model-file my_trained_model.bin --input-data-split-file my_data_split.bin --output-results-file my_results.json ``` The data …