VictorMuv/Kinyarwanda_model_repo

Starting modeling Kinyarwanda language as a pilot project. # Kinyarwanda Digitization Kinyarwanda digitization evokes a multiple of sub activities; Language to language translation, Text classifications, speech to text, text to speech, speech recognition, etc. The project that is currently ongoing under the sponsorship of GIZ limits itself on language to language translation or as mentioned machine translation. In close collaboration with Myculture and other relevant stakeholders; this machine translation project will use a specific but limited language model and Data to train this model is being collected. This was considered a pilot project for Kinyarwanda digitization. However; Kinyarwanda Digitization should enable other language related applications such as text categorization that have even much more interesting useful applications like; News grouping, sentiment analysis, hate speech recognition; englobed in Natural Language Processing or NLP. To give context to what NLP would be doing and what activities should take place; let’s begin by introducing some models and what it would take to prepare text or language datasets. * Character encoding This technique converts individual characters or letters into numbers or technically known as character tokenization. This technique, however has serious limitations because there are words (unigrams); that have the same characters in different order which a machine can’t recognize the difference. * Word Based encoding This techique; instead of tokenizing each single letter; it tokenizes full words. However better is compared to character encoding it still has shortcomings as it lacks semantic recognition. * Embedding This is the best technique of all; as it encodes sematics into vestors. This allows grouping related words by pointing their vestors in the same direction. ## The Concept After analysis of the written literature of the Kinyarwanda language; it was found that having Kinyarwanda model with the current written techniques it would be costly in terms of …