Text Normalization for Low-Resource Languages of Africa
Domain:
natural language processingRecord type:
paper
Training data for machine learning models can come from many different
sources, which can be of dubious quality. For resource-rich languages like
English, there is a lot of data available, so we can afford to throw out the
dubious data. For low-resource languages where there is much less data
available, we can't necessarily afford to throw out the dubious data, in case
we end up with a training set which is too small to train a model. In this
study, we examine the effects of text normalization and data set quality for a
set of low-resource languages of Africa -- Afrikaans, Amharic, Hausa, Igbo,
Malagasy, Somali, Swahili, and Zulu. We describe our text normalizer which we
built in the Pynini framework, a Python library for finite state transducers,
and our experiments in training language models for African languages using the
Natural Language Toolkit (NLTK), an open-source Python library for NLP.
Similar
Text Normalization for Low Resource Languagesgoogleinterns/text-norm-for-low-resource-languagesMassively Multilingual Text Translation For Low-Resource LanguagesAdversarial Text-to-Speech for low-resource languagesXF2T: Cross-lingual Fact-to-Text Generation for Low-Resource LanguagesText Image Generation for Low-Resource Languages with Dual Translation Learning