takenolab/kirundi-lemmatize-search

# KIRUNDI lemmatize and search ## Name KIRUNDI-nlp ## Description KIRUNDI-nlp is a tool to analyse Rundi texts by lemmatizing. That way NLP statistic tasks are more efficient, because it reduces tokens not only to types but to lemmata. This is important because in Bantu languages the verbform contains in itself also subject, up to three objects, time, negation, perfectiv and other features ... combined with 16 classes of nouns there is a huge number of forms possible. KIRUNDI-nlp takes a text and returns its tokens tagged with lemma and PoS. With this it's also possible: - to search N-grams by combinations of token, lemma or Part-of-Speech - view the frequency distribution of types - view the frequency distribution of lemmata - recieve a version of the text with lemmata replacing the tokens The tag will be "UNK" if it doesn't find an appropriate lemma or Named-Entity in the underlying database. ## Dependencies nltk, unidecode, pyqt It's written in Python3 ## Usage Run lemmatize_search/ingenzura_main.py Choose your preferred language for using the App: Rundi, English or German and start again. Follow the descriptions in the window in the right corner. After selecting the file you want to analyse, press "Lemmata" and you'll get some statistics and a csv-file with the tagged text, named "tag__filename.csv". In the same folder you'll find also the frequency distribution of lemmata of the text (named "fl__filename.csv") and a file with the normalised text (named "norm__filename.txt") and a version where all tokens are replaced by their lemma(this file is named lemma__filename.txt). Then enter your search-query. If there are results you'll find a list of maximum 20 results down in the window, but all results are stored in a txt-file. The name of the results-file ends with your search query. In our example it's: "/results/searched/Burundi_Culture_Nord-Est_-ri(lemma)_kugenda(lemma)_.txt". Each result includes a number followed by your search term with surroundi …