AUTOMATIC THESAURUS CONSTRUCTION FROM AFAAN OROMO TEXT USING WORD EMBEDDING

 Principal Advisor:  Gadisa Olani (PhD) ABSTRACT A Thesaurus is a reference work that enlists words grouped together according to similarity of meaning (containing synonyms and sometimes antonyms). One of the major problems of modern information retrieval systems is the vocabulary problem that concerns with the discrepancies between terms used for describing documents and the terms used by the searcher to describe their information need which forms the information overload or information mismatch. One way of overcoming with the vocabulary problem is using a thesaurus that shows the relationships between terms and query expansion which provides us the alternative terms for query to improve the effectiveness of retrieval. Since the manual thesaurus construction is a labour-intensive task and hence also expensive to build and hard to update in timely manner, Afaan Oromo automatic thesaurus is implemented by using the Word Embedding. In this study, we propose Automatic Afaan Oromo thesaurus construction using word embedding. The proposed model includes different tasks. The first task is text pre-processing which consists of commonly used text pre-processing tasks in many natural language processing applications. We perform text pre-processing in Afaan Oromo text document and train the document using a word embedding gensim library (word2vec) in order to generate word embedding model. The embedding result provides a contextually similar word for every word in the training set. The trained word vector model captures different patterns. After training the data we take the trained model as input and discover different patterns that used to extract thesaurus relations like: synonym, and antonym. Conceptual synonym of a word is extracted based on cosine similarity and Pearson correlation. We also used the Word2Vec model based on the configuration of windows size and different vector dimensions. So, for this method we perform Correlation task based on given sample seed words (synonym pairs).     A total of 1086133 words were collected and these collections is from different domains which covers different conceptual areas. Among these collected words 33836 vocabulary words were learned by the word2vec model. Our experimental measure using word2vec model and human judgment achieves 69.76% correlation. And We also evaluate using Spearman’s correlation, and achieve 82.85% correlation of the terms in the collection are registered to be similar as compare to human-rated similarities between words based on the configuration of windows size and different vector dimensions.