The pipeline employs a multi-layer scraping strategy to collect Tunisian Arabic tech QA pairs from Reddit.
It combines three data sources: the Reddit JSON API for real-time posts and comment threads, RSS feeds for post discovery with JSON
comment fetching, and Arctic Shift for historical data.
The system builds both direct QA pairs (post → top comment) and conversation chains (comment → reply) to capture nuanced technical discussions.