makhosi6/nguni

Looking into the digitizing of Nguni languages and increasing their digital footprint # nguni Looking into the digitizing of Nguni languages and increasing their digital footprint. ## Goals - The goal of this project is to come up with a - Language model - probability distribution over sequences of words - Large/comprehensive dataset - dictionary like document/resource that will spell out the use/meaning of each word/phrase in different contexts - detailing how it is used/mis-used, proper spelling and misspelling, pronunciation and alternative pronunciations how it has evolved overtime, its origin, etc - text-to-speech (TTS) model (_low priority_) - All documented in Nguni languages (_low priority_) - Text Analysis (_low priority_) ## Scope - To focus only on South African _Nguni languages_ (hence the name of the project), excluding Mozambican and Zimbabwean languages - isiZulu and isiXhosa as a starting point ## Vision Live it a world where I can - voice type in isiXhosa/isiZulu - get keyboard autocomplete in isiXhosa/isiZulu - and finally get over computers squiggly my name The bigger picture is to bring nguni culture and heritage to the modern world, and open doors to wide range of possibilities, such as - Closing the illiteracy/computer illiteracy gap by allowing everyone and anyone to access modern tools using their native languages - Making it possible to learn and teach in isiXhosa/isiZulu - Using isiXhosa/isiZulu to communicate at any level - Preserve and protect culture and heritage ### What it take * _Collect Language Data_: isiXhosa/Zulu language data to train your language model. This can include Xhosa/Zulu books, articles, news, and other text sources. You can also use publicly available datasets such as the South African National Corpus. * _Preprocess and Clean the Data_: This involves removing any unwanted characters, punctuation, and other non-text elements. * _Train a Language Model_: using PyTorch, TensorFlow, and Keras. * _Fine-tune the Model_: To improve the accuracy of your Xhosa/Zulu language model, you may nee …