googleinterns/text-norm-for-low-resource-languages

# Text Normalization for Low Resource Languages This repository contains code related to the Google open source internship project Text Normalization for Low Resource Languages. ## Project Description Training data for machine learning models can come from many different sources, which can be of dubious quality. For resource-rich languages like English, there is a lot of data available, so we can afford to throw out the dubious data. For low-resource languages where there is much less data available, we can’t necessarily afford to throw out the dubious data, lest we end up with a training set too small to train a model. The objective of this project was to study the effects of text normalization and data set quality for a set of low-resource languages of Africa. This involved building a text normalizer using Pynini, a Python library for finite state transducers, and training a language model using the Natural Language Toolkit (NLTK), an open-source Python library for NLP. ## Running the Pipeline ### Language Data The language-specific config files in `config/` include the paths to the language data files in `language_data/`. The repo does NOT include the data, however. You will need to download it yourself and make sure it is in the right place. The data we used comes from Universal Dependencies, the Leipzig Corpora Collection, OSCAR, and An Crúbadán. ### Running the Normalizer The text normalizer can normalize individual strings or load in a data file and normalize the whole file. The normalizer uses (up to) 5 flags - `language`: two-letter flag for the language, e.g. `af`, `mg`; exception is Bambara `bm_latn` - `data_source`: flag for the data source, e.g. `ud`, `lcc`, `ac`, `oscar`; not needed if you provide a value for `string_to_normalize` - `pass_valid`: whether to filter out individual tokens or sentences; value is `token` or `sentence` - `experiment`: name of experimental directory to create and output results to - `string_to_normalize`: a spec …