dohliam/more-stoplists

stoplists for African languages generated from the ASP corpus # more-stoplists - stoplists for African languages generated from the ASP corpus This project uses the source texts provided by the African Storybook Project as a corpus and provides a number of tools to extract frequency lists and lists of stopwords from this corpus for the 60+ languages covered by ASP. The immediate goal is to create freely-licensed stoplists for languages that are not currently included in the stopwords project, and to then submit those lists as pull requests to the upstream project. ## The problem The stopwords project currently includes stoplists for an impressive number of languages, however coverage of African languages is basically non-existent. Surprisingly, there appear to be no freely-available (let alone freely-licensed) lists of stopwords for even major African languages like Swahili anywhere online. (Articles like this one and this one mention creating Swahili stoplists, but the lists themselves are not available online in any form.) This was not an abstract problem, either. We wanted to filter stop words out of the ASP corpus to create lists of keywords for the ASP Imagebank Explorer and discovered that no existing project contained lists of stopwords for African languages. Like others before us we decided to create our own stoplists, but we also wanted to make them freely available so others would not have to keep reinventing the wheel this way in the future. ## A solution Fortunately, the African Storybook Project has been creating and translating short- and medium-form written texts in over 60 African languages and releasing them under an open license. The full corpus of these texts has been extracted and is now available in the ASP source repo, which makes it possible to provide reasonably good stoplists for many languages that are not currently covered by the stopwords project. ## Format The stoplists are arranged in folders according to ISO 639-1 or ISO 639-3 language code. So the Swahili stoplist is in the `sw` folder, …