Anaphase21/Probing-language-models-for-syntactic-and-semantic-knowledge-on-African-languages

Master's project where I probe language models for syntactic and semantic knowledge on African languages. # Probing-language-models-for-syntactic-and-semantic-knowledge-on-African-languages In this repo, I put my code, which I modified from another repo in order to run my experiments; a notebook of all the experiments; and a pdf document of my master's mini-thesis for this project. The code for this project was adapted from probing. The instructions remain the same as follows: # Probing Probing is a popular evaluation method for blackbox language models. In the simplest case, the representation of a token or a sentence is fed to a small classifier that tries to predict some linguistic label. This setup is exposed to a small amount of training data but only the classifier parameters are trained, the blackbox model's parameters are kept fixed. This library was developed for probing contextualized language models such as BERT. # Our papers that use this library This framework was used in the following projects: ## Subword Pooling Makes a Difference EACL2021 paper Github repository ## Evaluating Contextualized Language Models for Hungarian Paper Github repository ## Evaluating Transferability of BERT Models on Uralic Languages Paper Github repository # Probing types and tasks It supports two types of evaluation. ## Morphology probing Probe a single word in its sentence context, such as deriving the tense of the English word cut in these examples, where context clearly plays an important role: I cut my hair yesterday. Make sure you cut the edges. ### Data format Morphology probing uses TSV files as input. One line represents one sample. Each line has 4 tab-separated columns: 1. the full sentence 2. the target word 3. the index of the target word when using space tokenization 4. the label. The above two examples would look like this: ``` I cut my hair yesterday. cut 1 Past Make sure you cut the edges. cut 3 Pres ``` In **tagging tasks** each line represents a word and sentence boundaries are denoted by empty line …