dshumbanhete/Bootstrap_Shona_LM

Code implementation for my BComm (Hons) project at Rhodes University. Supervised by Dr. Shibeshi and Prof. Dlodlo, the study looks at bootstrap methods in NLP that can effectively model agglutinative languages under low-resource constraints. The model uses bootstrap ensemble sampling and bayesian variational inference in an RNN LM.. # Reducing Annotation Dependency in low-resource NER: a bootstrap approach to Shona Language Modelling Code implementation for my BComm (Hons) project at Rhodes University. Supervised by Dr. Shibeshi and Prof. Dlodlo, the study looks at bootstrap methods in NLP that can effectively model agglutinative languages under low-resource constraints. This research explores bootstrap techniques as a means by which to overcome the dependency on manual Annotation for Natural Language Processing (NLP) in Machine Learning (ML). The study focuses on Shona, an agglutinative language that is under-represented in ML. Unlike high-resource languages, African languages face data scarcity and agglutinative complexity, which limits the deployment of technology. The study employs bootstrapping, a statistical learning method that iteratively improves NLP models using minimal labelled training data. Chien and Ku (2015) claim that bootstrapping techniques can lend to task agnosticism, especially when coupled with the attention of a Bayesian LSTM. Using Bayesian Variational Inference in the weight calculations allows the model to effectively represent the relationships between the posterior and priors as variables/nodes in a bi-directional, cyclical graph. Results show that bootstrap techniques can help mitigate class imbalances. When combined with regularisation, bayesian models can effectively approximate the performance of a transformer while maintaining significantly shorter training times. NB: You will need to specify the data root and filepath for the files found in this repository that correspond to the code. Additionally, you will have to remove the dev data in mlm pretraining to reproduce the low-resource constraints quoted in the paper.