anrilombard/mzansi-text-tokenized
Domain:
natural language processingRecord type:
dataset
Creator:
anr
Host:
Ready-to-train tokenized version of MzansiText, chunked to a context length of 2048 tokens.
Tokenizer: custom BPE, 65536 vocabulary
Chunking: 2048 tokens per example with EOS separators between documentsSchema:
{
"input_ids": ["int"],
"lang": "string"
}
Split
Examples
Train
3,943,584
Validation
19,379
Test
19,341
from datasets importload_dataset