JalenJohnsonB/ht-nlp-thesis

This is the codebase for The Impact of Genre on Nasalization and Shortened Forms in Haitian Creole: Exploratory Work on Natural Language Processing for Low-Resource Languages, submitted by Jalen Johnson in partial fulfillment of the requirements for the degree of Bachelor of Arts in the Department of Computer Science, Princeton University This is the codebase for The Impact of Genre on Nasalization and Shortened Forms in Haitian Creole: Exploratory Work on Natural Language Processing for Low-Resource Languages, submitted by Jalen Johnson in partial fulfillment of the requirements for the degree of Bachelor of Arts in the Department of Computer Science, Princeton University === Overview === Run train_ht_stanza.sh to train charlm, pos tagger, and lemmatizer on the baseline corpus (found in /corpora_academia) and the augmented corpus (/corpora_academia + /raw_ht_text), will output saved models to /saved_models token_type_counts.txt -- token/type rundown of both corpora split_conllu.py -- splits ht_autogramm-ud-test.conllu for train/dev/test prepare_charlm_data.py -- splits corpora .txt files into train/dev make_pretrain.py -- makes .pt files from prepare_charlm_data.py output Each script has a general comment at the top detailing the general function of that file, its input/outputs, and how to run it in the command line. Additional config files and helper scripts are detailed below === Config Files === genre_config_aug.json -- lists genres of all .txt files in /raw_ht_text genre_config_base.json -- genres of all .txt files in /corpora_academia === Helper Scripts === diagnose_lid.py -- runs fasttext langID on a Haitian Creole .txt file to identify a useful confidence score to use filter_ht_fasttext.py -- cleans .txt files, removes non-Haitian Creole (uses confidence score from diagnose_lid.py) scan_ht.py -- scans both corpora (/corpora_academia, /raw_ht_text) for linguistic phenomena stats and outputs analysis to analysis_output as .csv graph_ht.py -- generates graphs from .csv files in analysis_output (uses data generated from scan_ht.py) === Notes === This work uses cloned Stanza files to train models in the pipeline, found in the /stanza folder, and may be degraded. To clone Stanza files to have an updated /stanza folder, run: git clone github.com This work us …