TunisIA-Co-Lab/reddit_scraper

Multi-layer pipeline for scraping Tunisian Arabic tech QA pairs from Reddit without API authentication. # TunisIA Data Pipeline Modular, reproducible pipeline for building the **Reddit Tunisia Tech QA** dataset from Reddit posts and comments. ## File Structure & Roles ``` reddit_scraper/ ├── config/ Configuration management │ ├── settings.py Central config: file paths, thresholds, keywords, dataset size targets │ └── __init__.py Package initializer │ ├── utils/ Reusable utilities │ └── helpers.py HTTP helpers, PII cleaning, dialect detection, quality scoring functions │ ├── stages/ Pipeline stages (data flow) │ ├── 01_scrape.py [INPUT: Reddit API] Collect raw Q&A pairs from JSON API, RSS, Arctic Shift │ ├── 02_filter.py [INPUT: raw.jsonl] Dedup, re-score, filter, rank → clean dataset │ ├── 03_nsfw_filter.py [INPUT: reddit_tunisia_qa.jsonl] Remove NSFW/harmful content, bad words │ ├── 03_inspect.py Generate quality reports, flags, statistics on dataset │ └── 04_upload_hf.py [OUTPUT: HuggingFace Hub] Push cleaned dataset with README │ ├── ranking/ Alternate pipeline structure (for reference) │ ├── 03_inspect.py │ └── 04_upload_hf.py │ ├── output/ Generated datasets & reports (created automatically) │ ├── raw.jsonl Raw scraped pairs (no filtering) │ ├── filtered.jsonl After deduplication & ranking │ ├── reddit_tunisia_qa.jsonl Final dataset before NSFW filtering │ ├── reddit_tunisia_qa_clean.jsonl Final cleaned dataset (NSFW/bad words removed) │ ├── scrape_stats.json Scraping metadata & counts │ ├── quality_report.json Quality score breakdown, flags, percentiles │ └── nsfw_filter_stats.json NSFW filtering statistics │ ├── run_pipeline.py Orchestrator: runs stages sequentially or individually ├── requirements.txt Python dependencies └── README.md This file ``` ## Running the Pipeli …