chrisjay/ratman-stopword-lists-for-african-languages

Some words, like “the” or “and” in English, are used a lot in speech and writing. For most Natural Language Processing applications, you will want to remove these very frequent words. This is usually done using a list of “stopwords” which has been complied by hand.

This project uses the source texts provided by the African Storybook Project as a corpus and provides a number of tools to extract frequency lists and lists of stopwords from this corpus for the 60+ languages covered by ASP.

Included in this dataset are the following languages:

• Afrikaans: stoplist and word frequency
• Hausa: stoplist and word frequency
• Lugbarati: word frequency only
• Lugbarati (Official): word frequency only
• Somali: stoplist and word frequency
• Sesotho: stoplist and word frequency
• Kiswahili: stoplist and word frequency
• Yoruba: stoplist and word frequency
• isiZulu: stoplist and word frequency

Files are named using the language’s ISO code. For each language, code.txt is the list of stopwords, and code_frequency_list.txt is word frequency information. A list of ISO codes the the languages associated with them may be found in ISO_codes.csv.