ajoer/aave_pos

POS tagging for African-American Vernacular English # POS tagging for African-American Vernacular English This is part of my Master's thesis in 2015. The project aimed at improving Part-of-Speech (PoS) tagging for an under-resourced language version, African-American Vernacular English (AAVE). I used unsupervised and weakly (or semi-)supervised machine learning methods to improve the accuracy of an out of the box, state-of-the-art PoS tagger, the Rungsted tagger. The main issue with the tagging accuracy on AAVE data is that there are many out-of-vocabulary (OOV) words. This is due to the dialect, in which word choices can differ from General American, but it is also due to spelling variations in the dialect. Since PoS tagging is conducted on written language only, any variation in spelling has a large impact on the tagging accuracy. In order to overcome this issue, I want to add annotated AAVE data to the training data of the PoS tagger model, such that the tagger learns the variations in word choice and spelling, and the number of OOV words decreases. However, since AAVE is an under-resourced language version, meaning there are no annorated data resources to give to out model, I have to create these annotated resources. I want to annotate the data automatically, since I need a lot of it to overcome the data bias of the General American data. I need two data sources; unlabelled AAVE data and some sort of linguistic information about the OOV words that can help me annotate the unlabelled data. For unlabelled data I use tweets written in the American southeastern states, subtitles from the TV shows The Wire and Boondocks and hiphop lyrics from African-American rappers. I restrict the tweets to this geographical area, since the proportion of AAVE speakers is highest here and I only include tweets that contain at least one AAVE spelled word. See Jorgensen et al. 2015 for more on this approach. For linguistic information, I create tag dictionaries using Urban Dictionary, Wiktionary, Hepster's and a list of African- …