South African Languages Dataset

Language Training Documents Training GPT2 Tokens Avg Tokens/Doc Max Tokens Test Documents Test GPT2 Tokens Test Avg Tokens/Doc Test Max Tokens isiZulu 116,693 192,622,799 1,650.68 335,530 687 1,080,961 1,573.45 15,691 Sesotho 83,329 144,337,938 1,732.15 98,542 841 1,393,086 1,656.4614,071 isiXhosa 99,567 141,484,241 1,421.00 113,710 788 1,161,296 1,473.73 17,220 isiNdebele 21,922 17,533,799 799.83