Felladrin/ChatML-aya_dataset
Domaine:
natural language processingType de record:
dataset
Créateur:
Fel
Hôte:
CohereForAI/aya_dataset in ChatML format, ready to use in HuggingFace TRL's SFT Trainer.
Python code used for conversion:
from datasets import load_dataset
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("Felladrin/Llama-160M-Chat-v1")
dataset = load_dataset("CohereForAI/aya_dataset", split="train")
def format(columns):
messages = [
{
"role": "user",
"content": columns["inputs"].strip(),
},