Oyebamiji-Micheal/Yoruba-Word-Representations-using-FastText

Yoruba Word Representations and Semantic Visualization using FastText and UMAP # Yoruba Word Representations and Semantic Visualization using FastText and UMAP In this repo, I trained FastText word embeddings on a Yorùbá corpus and then visualize the resulting semantic space using UMAP. I have been doing a lot of learning lately on Natural language processing, especially for low resource language. Part of my learning is reimplementing most NLP concepts. Today, my goal is to train a simple model to represent semantics. Note: This repository is not a tutorial nor an article and I don't expect anyone to treat it as such. They are just for documenting my learning journey of each NLP concept. ## Dataset I have read this Alabi's paper (see below) several times early this year and I come across it when I read most studies on NLP. Of course, I am not trying to recreate the whole paper. All I want to do is gather tiny bits of knowledge from every crane. One of the datasets used in the paper is denoted as C3 which aggregates Yorùbá text from sources like the Bible, JW300, Wikipedia, BBC Yorùbá and VON Yorùbá. It has both clean text with proper diacritics and noisier text from sources where diacritics are often missing. See citation at the bottom. ## Why FastText? FastText learns representations at the subword level using character n-grams. This matters a lot for Yorùbá because the language is tonal and morphologically rich. Small changes in diacritics change meaning entirely (`ọjà` = market, `ọja` = rope). The character n-gram approach lets the model share information across words that share roots, which helps a lot when your corpus is small. I trained with the skipgram objective which tends to produce better representations for rare words, again useful in a low-resource setting. ## What the embeddings actually learned Most of the clustering is driven by character-level similarity rather than pure semantics, which is expected. But there are genuinely interesting groupings in there too. `ọjà` (market) pulls in `pọlọ́wọ̀` and `ìpọlọ́wọ̀` (ad …