bahaeddinmselmi/derja-smart-scraper

A lightweight CLI tool for collecting Tunisian Derja text snippets from the open web. It queries Google via [SerpAPI](serpapi.com), downloads each result, extracts readable text, and keeps only the sentences that look like Tunisian Derja using a heuristic detector. # 🇹🇳 Derja Smart Web Scraper > A specialized web mining tool for **Tunisian Arabic (Derja)**, designed to build high-quality NLP datasets for LLM training. ## 📖 Overview **Derja Smart Web Scraper** is a lightweight CLI utility that automates the collection of Tunisian Arabic text from the open web. It leverages **Google Search (via SerpAPI)** to discover authentic content, scrapes the resulting pages, and applies a heuristic **Derja Detector** to filter for relevant sentences. It is specifically engineered to handle the nuances of the Tunisian dialect, including: * **Code-switching**: Mixing French/English with Arabic. * **Arabizi**: Latin-script Arabic (e.g., "chnowa a7welek"). * **Arabic Script**: Standard and dialectal Arabic text. ## ✨ Key Features * **Intelligent Search**: Automates Google queries to find niche forums and blogs. * **Smart Filtering**: Uses a vocabulary-based scoring system (`score_derja`) to distinguish Tunisian Derja from MSA (Modern Standard Arabic) or other noise. * **Clean Output**: Produces ready-to-train **JSONL** data, compatible with standard NLP pipelines. * **Robust Cleaning**: Strips HTML, ads, and navigation elements using BeautifulSoup. ## 🛠 Installation 1. **Clone the repository**: ```bash git clone github.com cd derja-smart-scraper ``` 2. **Install dependencies**: ```bash pip install -r requirements.txt ``` 3. **Set up API Key**: You need a SerpAPI key to perform searches. ```bash # PowerShell $env:SERPAPI_API_KEY = "YOUR_API_KEY" ``` ## 🚀 Usage Run the scraper with custom queries to start building your dataset: ```bash python -m smart_scraper.collect_web_derja_search \ --queries "كلام تونسي" "proverbes tunisiens" "nokat tounsiya" \ --max_results_per_query 20 \ --min_score 0.15 \ --out_file data/my_tunisian_dataset.jsonl ``` ### Options | Flag | Description | Default | |------|-------------|---------| | `--queries` | List of search terms | (Interna …