Si-Rin/Tunisian-data-collection

Tunisian social media comments — a text-based dataset labeled for sociological analysis (positive, negative, neutral/mixed). # Tunisian Arabic Data Collection A scalable data collection and merging pipeline for building datasets aimed at **Tunisian Arabic emotion/sentiment detection**. Collects social media comments from YouTube, Twitter/X, and Reddit, then merges them with existing labeled datasets (ArPanEmo, ArSAS, TSAC) into a unified training set. --- ## Table of Contents - Overview - Project Structure - Installation - Configuration - Usage - Collecting Data - Processing Collected Data - Merging All Datasets - Label Schema - Data Sources - Known Issues & Limitations - Contributing --- ## Overview This project addresses the scarcity of labeled Tunisian Arabic (Tunisian dialect / Darija) data for NLP tasks. It provides: - **Collectors** for YouTube comments, Twitter/X tweets, and Reddit posts/comments - **Two collection modes** per platform: by specific ID or by keyword search - **A merge pipeline** that combines collected (unlabeled) data with three existing labeled datasets into a single Excel file ready for annotation or training ``` Social Media APIs ├── YouTube Data API v3 ──┐ ├── Twitter API v1.1 ──┼──► JSONL files ──► collected_dataset.xlsx └── Reddit API (PRAW) ──┘ Labeled Datasets ├── ArPanEmo (.xlsx) ──┐ ├── ArSAS (HuggingFace) ──┼──► merged_dataset.xlsx └── TSAC (HuggingFace) ──┘ + collected_dataset ``` --- ## Project Structure ``` tunisian-data-collection/ │ ├── src/ │ ├── collect.py # Main CLI entry point for data collection │ ├── fix_jsonl.py # Utility to repair malformed JSONL files │ ├── merge_datasets.py # Merges all sources into final dataset │ │ │ ├── collectors/ │ │ ├── __init__.py │ │ ├── base_collector.py # Abstract base class for all collectors │ │ ├── youtube_collector.py # YouTube Data API v3 collector │ │ ├── twitter_collector.py # Twitter API v1.1 collector │ │ └── reddit_collector.py …