bahaeddinmselmi/tunisian-arabic-ai-dataset

The largest open-source dataset for Tunisian Arabic (Derja) NLP, featuring social media text, transcripts, and e-commerce data for LLM training and fine-tuning. # 🇹🇳 Tunisian Arabic (Derja) AI Dataset > The largest open-source collection of Tunisian Arabic text for NLP, LLM training, and sentiment analysis. ## 📖 Overview The **Tunisian Arabic AI Dataset** is a comprehensive corpus aggregated from multiple social media platforms and online sources. It is designed to facilitate the development of Large Language Models (LLMs), chatbots, and translation systems for the **Tunisian dialect (Derja/Tounsi)**, which is a low-resource language variety often mixing Arabic script and "Arabizi" (Latin script). This repository hosts a consolidated **334 MB+** archive containing over **170 data files**, ready for machine learning pipelines. ## 📦 Dataset Contents The data is provided in a single Git LFS-tracked archive: **`tunisian_datasets.zip`**. ### 1. Social Media Corpus Extensive scrapes from major social platforms, capturing authentic, informal, and code-switched communication: * **Reddit (r/Tunisia)**: * `reddit_posts_all.jsonl`: Full historical post data. * `reddit_comments_all.jsonl`: Deep comment threads, rich in Arabizi and English-Derja mixing. * **Facebook & Twitter/X**: * Scraped posts and comments from public Tunisian communities. * High density of native Arabic script Derja. * **YouTube**: * Transcripts and comment sections from popular Tunisian channels. ### 2. Pre-Processing & Training Splits Ready-to-use splits for Supervised Fine-Tuning (SFT) and Evaluation: * **`cleaned.jsonl`**: Text normalized to remove PII, excessive noise, and non-text artifacts. * **`dedup.jsonl`**: Determining deduplicated content to ensure model quality. * **Train/Test/Val Splits**: * `train.jsonl` (~80%) * `val.jsonl` (~10%) * `test.jsonl` (~10%) ### 3. Specialized Sub-Datasets * **Lexicons**: `TuniziDataset.csv` for sentiment analysis and dialect classification. * **E-Commerce**: `products.json` and mock conversations for building commercial RAG (Retrieval-Augmented Generation) agents. ## 🚀 Getting …