DeepsMoseli/TswanaBert

TswanaBERT is a transformer model pre-trained on a corpus of Setswana language data in a self-supervised fashion. works in both pytorch and TF --- language: setswana --- # TswanaBert ## Model Description. TswanaBERT is a transformer model pretrained on a corpus of Setswana data in a self-supervised fashion by masking part of the input words and training to predict the masks. ## Intended uses & limitations The model can be used for either masked language modeling or next word prediction. it can also be fine-tuned for a specifict application. #### How to use ```python >>> from transformers import pipeline >>> from transformers import AutoTokenizer, AutoModelWithLMHead >>> tokenizer = AutoTokenizer.from_pretrained("MoseliMotsoehli/TswanaBert") >>> model = AutoModelWithLMHead.from_pretrained("MoseliMotsoehli/TswanaBert") >>> unmasker = pipeline('fill-mask', model=model, tokenizer=tokenizer) >>> unmasker("Ntshopotse e godile.") [{'score': 0.32749542593955994, 'sequence': ' Ntshopotse setse e godile. ', 'token': 538, 'token_str': 'Ġsetse'}, {'score': 0.060260992497205734, 'sequence': ' Ntshopotse le e godile. ', 'token': 270, 'token_str': 'Ġle'}, {'score': 0.058460816740989685, 'sequence': ' Ntshopotse bone e godile. ', 'token': 364, 'token_str': 'Ġbone'}, {'score': 0.05694682151079178, 'sequence': ' Ntshopotse ga e godile. ', 'token': 298, 'token_str': 'Ġga'}, {'score': 0.0565204992890358, 'sequence': ' Ntshopotse, e godile. ', 'token': 16, 'token_str': ','}] ``` #### Limitations and bias The model is trained on a fairly small collection of setwana, mostly from news articles and creative writtings, and so is not representative enough of the language as yet. ## Training data 1. The largest portion of this dataset (10k) sentences of text, comes from the Leipzig Corpora Collection 2. I Then added SABC news headlines collected by Marivate Vukosi, & Sefara Tshephisho, (2020) that is generously made available on zenoodo. This added 185 tswana sentences to my corpus. 3. I went on to add 300 more sentences by scrapping following sites news sites and blogs that mostly based in Botswana. I actively con …