Language
Training Documents
Training GPT2 Tokens
Avg Tokens/Doc
Max Tokens
Test Documents
Test GPT2 Tokens
Test Avg Tokens/Doc
Test Max Tokens
isiZulu
116,693
192,622,799
1,650.68
335,530
687
1,080,961
1,573.45
15,691
isiXhosa
99,567
141,484,241
1,421.00
113,710
788
1,161,296
1,473.7317,220
isiNdebele
21,922
17,533,799
799.83
42,701
222
170,111
766.27
6,615
siSwati
1,668
3,148,007
1,887.29
24,129