The Challenge of Diacritics in Yoruba Embeddings

Domain:

natural language processing

Record type:

paperdataset

Creator:

AdeLiwLiw

Host:

The major contributions of this work include the empirical establishment of a better performance for Yoruba embeddings from undiacritized (normalized) dataset and provision of new analogy sets for evaluation. The Yoruba language, being a tonal language, utilizes diacritics (tonal marks) in written form. We show that this affects embedding performance by creating embeddings from exactly the same Wikipedia dataset but with the second one normalized to be undiacritized. We further compare average intrinsic performance with two other work (using analogy test set & WordSim) and we obtain the best performance in WordSim and corresponding Spearman correlation. Presented at NeurIPS 2020 Workshop on Machine Learning for the Developing World

bumie-e/Yoruba-diacritics-vs-non-diacritics

Professor/yoruba-diacritics-quantized

YasinProDebian/yoruba-diacritics-quantized

Olamyy/yoruba-embeddings

A collection of Yoruba Embeddings

YORNET - Visualizing word embeddings for the Yoruba language

This is a visualization of the yoruba language word embeddings, using T-SNE, a work conducted by our NLP research engineers at Rectlabs Inc, who are working on language modelling of the yoruba language. This is ongoing and we expect that the model gets better with

freshiwe/On-the-Surface-of-Tone-What-Multilingual-LLMs-Do-and-Don-t-Know-About-Yoruba-Diacritics

Research paper titled: On the Surface of Tone: What Multilingual LLMs Do and Don't Know About Yoruba

The Challenge of Diacritics in Yoruba Embeddings

Domain:

Record type:

Visit

Tasks

Languages