dsfsi/PuoData
Domain:
natural language processingRecord type:
dataset
Creator:
dsf
Host:
Curated corpora for Setswana. Used to train PuoBERTa.
# PuoData: A curated corpora for Setswana
Give Feedback đź“‘: DSFSI Resource Feedback Form
We believe that PuoData is a valuable resource for the Setswana language community. We hope that PuoData will be used to develop new and innovative applications that benefit the Setswana-speaking community.
## Dataset Curation
| Dataset Name | Kind | Num. of Tokens |
|---|---|---|
| *PuoData* | | |
| NCHLT Setswana \cite{eiselen2014developing} | Government Documents | 1,010,147 |
| Nalibali Setswana | Childrens Books | 57,654 |
| Setswana Bible | Book(s) | 879,630 |
| SA Constitution | Official Document | 56,194 |
| Leipzig Setswana Corpus BW | Curated Dataset | 219,149 |
| Leipzig Setswana Corpus ZA | Curated Dataset | 218,037 |
| SABC Dikgang tsa Setswana FB (Facebook) | News Headlines | 167,119 |
| SABC MotswedingFM FB | Online Content | 33,092 |
| Leipzig Setswana Wiki | Online Content | 230,333 |
| Setswana Wiki | Online Content | 183,168 |
| Vukuzenzele Monolingual TSN | Government News | 157,798 |
| gov-za Cabinet speeches TSN | Government Speeches | 591,920 |
| Department Basic Education TSN | Education Material | 708,965 |
| **PuoData Total** | 25MB on disk | **4,513,206** |
| *PuoData+JW300* | | |
| JW300 Setswana| Book(s) | 19,782,122 |
| **PuoData+JW300** | 124MB on disk | **24,295,328** |
## Dataset Uses
We used this corpus to train PuoBERTa, 🤗 PuoBERTa. It is also part of the corpus used for PuoBERTaJW300.
## Citation Information
Bibtex Reference
```
@inproceedings{marivate2023puoberta,
title = {PuoBERTa: Training and evaluation of a curated language model for Setswana},
author = {Vukosi Marivate and Moseli Mots'Oehli and Valencia Wagner and Richard Lastrucci and Isheanesu Dzingirai},
year = {2023},
booktitle= {SACAIR 2023 (To Appear)},
keywords = {NLP},
preprint_url = {arxiv.org,
dataset_url = {github.com,
software_url = {huggingface.co
}
```
# …