ctaguchi/lowresource_postag_crf

POS tagging with CRF for low-resource languages on Universal Dependencies # lowresource_postag_crf This is a simple CRF-based POS tagger for low-resource languages using data from Universal Dependencies. This repository is prepared for the presentation at the Linguistics Festival 2022. ## Requirements Check the required packages in `requirements.txt`. `sklearn_crfsuite` is not included in the default Python3. An easy way to install it is just to type on the command line `pip install sklearn_crfsuite`. The detailed documentation and friendly tutorial can be found here. ## Usage ### Clone the repository Clone this repository to your local environment. To do so, on your command line, move to the directory where you want to work on, and type `git clone github.com`. ### Prepare data To train a model, you need to have a dataset with the CoNLL-U format. An easy way to get data is just to go to Universal Dependencies and go to a repository of a language that you want to test on. After downloading the data, I recommend you to place the data in the same directory that you have just cloned, because it will make it easy later for you to input file names. ### Preprocess data The raw data do not come in the way we want, because it may have unnecessary information or ill-formed parts. `preprocess.py` takes the UD-format raw file that you have just downloaded and creates a training data and a test data in the `.csv` format, which is often suitable for machine learning. Immediately after running `python preprocess.py`, you are asked to write the file name of the raw data (`.conllu` file). Then, you are asked to write the file name that you want as the output data (Note that the name has to end with `.csv`). You can change the ratio of train-test splitting by changing the value of the `ratio` variable. ### Train the model `crf.py` defines the architecture of CRF, including features, and the evaluation. After running `python crf.py`, you are asked to write the file name of the training data. Then, you …