anrilombard/mzansi-text-tokenized
Domaine:
natural language processingType de record:
dataset
Créateur:
anr
Hôte:
Ready-to-train tokenized version of MzansiText, chunked to a context length of 2048 tokens.
Tokenizer: custom BPE, 65536 vocabulary
Chunking: 2048 tokens per example with EOS separators between documentsSchema:
{
"input_ids": ["int"],
"lang": "string"
}
Split
Examples
Train
3,943,584
Validation
19,379
Test
19,341
from datasets importload_dataset