leobitz/amharic_word_embeddings

# amharic word embedding resources Here you will find resources for amharic word embedding ## Corpus I have collected a corpus from most of the amharic news websites. After cleaning, the dataset contains around 37.9 million tokens with 1.39 million unique tokens. You can find the uncleaned raw dataset here: Raw Dataset You can find the cleaned dataset here: Clean Dataset ## Cleaning I cleaned the dataset using the following procedure. But, you can apply your own on the raw dataset. * Normalize by replacing characters and words by using the mapping in the replace file * Replaced consecutive same punctuations by just one. Example: %%% -> % * Added spaces around punctuations. Example: ቻው! -> ቻው ! * Truncated words that have more than 13 characters to just 13 * Replaced characters other than arabic digits and the characters in charset with 'u'. Example: እንሂድxc -> እንሂድuu * Replaced words that are not amharic with 'unk'. Example: she said ልክ ነው -> unk unk ልክ ነው * Replaced consecutive 'unk' with just one 'unk'. Example: she said ልክ ነው -> unk ልክ ነው ## Alphabetized Word Embeddings Please read our paper to understand why Alphabetized word embeddings are better than the normal fasttext word embeddings. In short, it unravels the syntactic information between characters. That simply allows the fasttext algorithm to learn better embeddings that captures syntactic information. Example of alphabetization: ትሄዳለች -> ተæሀêደaለቸæ To convert an amharic text file to alphabetized one, use this script ## Amharic Word Analogy In our study, we collected an amharic word analogy test. You can find it here. If you need the alphabetized version, find it here ## Embeddings All these word embeddings are trained on the Clean Dataset. For word2vec, the window is set to 5, which is the default. For the fasttext embeddings however, it is set to 1 as the performance drops as the window size grows. | | Word2vec | Fasttext | Alphabetized Fas …