essiaajroud/tunisian_story_scraper

A Python web scraper that extracts Tunisian stories, legends, and folk tales from turathy.tn. The scraper collects content across multiple categories including myths and legends (خرافات و أساطير) and stories and tales (قصص و حكايات). # Tunisian Story Scraper A Python web scraper that extracts Tunisian stories, legends, and folk tales from turathy.tn. This project preserves and collects traditional Tunisian cultural content across multiple categories, with output in multiple formats (JSON, CSV, TXT). ## 📋 Project Overview This scraper automates the collection of Tunisian heritage content from the Turathy platform, handling: - Multiple content categories - Arabic text processing and encoding - Intelligent content extraction and filtering - Data output in three formats for maximum compatibility ## ✨ Features - 🎯 **Smart Content Extraction**: Uses multi-stage strategy to identify and extract main article content - 🌐 **Multi-Category Support**: - **Myths and Legends** (خرافات و أساطير) - **Stories and Tales** (قصص و حكايات) - 🛡️ **Robust Error Handling**: Fallback mechanisms for unavailable pages - 📦 **Multi-Format Output**: - **JSON** - Structured data format - **CSV** - Spreadsheet-compatible format - **TXT** - Human-readable format with metadata - 🔤 **Arabic Text Support**: Full UTF-8 encoding for Arabic content - ⏱️ **Rate Limiting**: Respectful delays between requests (2 seconds default) - 🤖 **User-Agent Spoofing**: Realistic browser headers to avoid blocking ## 🛠️ Technologies & Tools ### Core Libraries | Tool | Purpose | Version | |------|---------|---------| | **requests** | HTTP client for fetching web pages | Latest | | **BeautifulSoup4** | HTML parsing and DOM traversal | 4.x | | **Python** | Core programming language | 3.7+ | ### Built-in Modules - **json** - JSON serialization and output - **csv** - CSV file generation - **re** - Regular expressions for text cleaning and URL removal - **time** - Request scheduling and rate limiting - **os** - File path management - **urllib.parse** - URL processing and joining ## 📋 Architecture ### Key Components #### 1. **URL & Configuration Management** ```python BASE_URL = "turathy.tn" LISTING_PAGES = { "kharafat": "...", # M …