LauraMartinus/ukuxhumana

Neural Machine Translation for South African Languages # Ukuxhumana "Ukuxhumana" means "Communicate" in Zulu. This project is aimed at exploring ideas for using Neural Machine Translation for low-resource languages - right now, specifically for the official languages of South Africa, but we are looking for collaborators across the continent to work together with us for the other languages # Mission - Provide a centralized repository for known datasets for African NMT and other NLP applications. - Provide pretrained state-of-the-art models for African languages. - Decrease the barrier to doing NMT research for African languages by providing code and data and models. - Spur collaboration across the continent to work on these problems together. # Data ## Parallel Corpuses Our parallel corpuses are from the Autshumato project. The datasets contain data that was translated by professional translators, data that was sourced as translated file pairs from translators and data obtained from Government websites and documents. We also performed extra cleaning on the corpuses, which is described here ## Monolingual Corpuses Our monolingual corpuses are from a variety of sources. We've used the monolingual corpuses for use in the training of fastText embeddings, which are also used in Unsupervised NMT. ### Zulu - Leipzig Zulu 100K Corpus - NCHLT isiZulu Text Corpora cleaned by Bernhard Duvenhage ### English - WMT 2014 ## Known Corpuses We keep a list of known corpuses for African languages here. Please consider contributing a link to your corpus :) # Models Currently, two main architectures are used throughout this project, namely Convolutional Sequence to Sequence by Gehring et. al. (2017) and Transformer by Vaswani et. al (2017). Fairseq(-py) and Tensor2Tensor were used in modeling these techniques respectively. For each language, a model was trained using byte-pair encoding (BPE) for tokenisation. The learning rate was set to 0.25 and dropout to 0.2. Beam search with a width of 5 was used in decoding the test data …