ImadSaddik/BoDmaghDataset

BoDmagh dataset is a Supervised Fine-Tuning (SFT) dataset for the Darija language # BoDmagh dataset 🧠 The BoDmagh dataset is a Supervised Fine-Tuning (SFT) dataset for the Darija language. I created it manually, ensuring high quality. The dataset is in JSON format and includes conversations between a user and an assistant. You can find the dataset on Hugging Face: here I update the dataset daily, so make sure to check the repository regularly. ## Time spent Creating this dataset has been a labor of love. I’ve dedicated approximately **19 hours and 10 minutes** so far, manually curating each entry to ensure high quality. I continue to add new entries daily, so this number will keep growing. ## Content ### 1. Structure The dataset is a JSON file containing a list of objects. Each object represents a single conversation entry and includes several metadata fields along with the conversation itself. Here are the main fields within each conversation object: * **`id`**: A unique integer identifier for the conversation. * **`conversation`**: A list of message objects representing the turns in the conversation. Each message object has: * `role`: Either `user` or `assistant`. * `content`: The text content of the message turn (plain text). * **`token_count`**: The total number of tokens in the `conversation` field, calculated using the darija tokenizer. * **`turns_count`**: The total number of turns (messages) in the `conversation`. * **`markdown_conversation`**: A list of message objects similar to `conversation`, but the `content` may include Markdown formatting (e.g., bolding, lists) where applicable. * **`source`**: Indicates how the conversation was generated (e.g., "Manually generated"). * **`topic`**: A label indicating the primary topic of the conversation (e.g., "Science", "Travel", "Chit-chat/Games/Humor"). * **`safety_flags`**: A list of strings indicating any safety concerns identified in the conversation (e.g., `["harmful_content_request", "refusal"]`). An empty list `[]` means the conversation is considered safe. *Note: The `system` …