natitam1/amharic-sentence-scraper

# Amharic Sentence Scraper Amharic Sentence Scraper is an open-source Python pipeline that discovers Amharic news articles and converts them into clean, standalone sentences for language-model and text-to-speech text datasets. It writes UTF-8 JSONL, records source attribution, estimates duration through word count, removes exact and near duplicates, respects `robots.txt`, rate-limits each domain, and remembers completed articles so scheduled runs can resume safely. This repository contains software, not a redistributed news corpus. Users are responsible for reviewing every source's current terms, robots policy, copyright restrictions, and permitted data uses. ## Features - Amharic-aware boundaries for `።`, `፡፡`, `!`, and `?` - Configurable 5 to 90 word default range - Ethiopic Unicode script-ratio validation - Boilerplate, link, email, hashtag, and fragment rejection - Exact SHA-256 and bounded near-duplicate detection - RSS, sitemap, and category-page URL discovery - Configurable HTML body and removal selectors - Per-domain delays, retries, timeouts, and robots enforcement - JSONL records with source, URL, dates, word count, and character count - SQLite state for article and sentence idempotency - Per-source summaries and random QA samples - Shared adapter contract for future captions and transcripts ## Requirements - Python 3.11 or newer - Internet access for live collection - Disk space for JSONL output and SQLite state - A real contact address in the crawler User-Agent ## Installation ```bash git clone github.com cd amharic-sentence-scraper ``` Windows PowerShell: ```powershell python -m venv .venv .\.venv\Scripts\Activate.ps1 python -m pip install --upgrade pip python -m pip install -r requirements.txt ``` Linux or macOS: ```bash python3 -m venv .venv source .venv/bin/activate python -m pip install --upgrade pip python -m pip install -r requirements.txt ``` Verify the installation: ```bash python main …