South African Nguni Languages Dataset

Language Training Documents Training GPT2 Tokens Avg Tokens/Doc Max Tokens Test Documents Test GPT2 Tokens Test Avg Tokens/Doc Test Max Tokens isiZulu 116,693 192,622,799 1,650.68 335,530 687 1,080,961 1,573.45 15,691 isiXhosa 99,567 141,484,241 1,421.00 113,710 788 1,161,296 1,473.7317,220 isiNdebele 21,922 17,533,799 799.83 42,701 222 170,111 766.27 6,615 siSwati 1,668 3,148,007 1,887.29 24,129