Felladrin/ChatML-aya_dataset

CohereForAI/aya_dataset in ChatML format, ready to use in HuggingFace TRL's SFT Trainer. Python code used for conversion: from datasets import load_dataset from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("Felladrin/Llama-160M-Chat-v1") dataset = load_dataset("CohereForAI/aya_dataset", split="train") def format(columns): messages = [ { "role": "user", "content": columns["inputs"].strip(), },