haider0708/NINJA-DERJA-SCRAPPER

NINJA DERJA SCRAPPER # NINJA DERJA SCRAPER Professional terminal scraper for Derja.Ninja search results, focused on Tunisian Arabic / `بالتونسي` queries. It reads words or sentences, scrapes the top Derja.Ninja search samples, writes clean JSONL + manifest files, and can optionally download and trim timestamped audio clips when you have permission to use the recordings. ## Quick Start Install locally: ```powershell .\.venv\Scripts\python.exe -m pip install -e ".[dev]" ``` Put your words in the fixed input file: ```text input/input.txt ``` One query per line: ```text برشا سلام شنوة ``` Run the batch scraper: ```powershell ninja-derja scrape ``` The CLI uses `input/input.txt` by default, runs with parallel workers, and shows progress with remaining time in real terminals. ## Interactive CLI Open the branded terminal app: ```powershell ninja-derja ``` From there you can: - scrape one word or sentence - scrape from `input/input.txt` or another text file - edit config values from the terminal - review the latest output summary Arabic input is handled with a custom right-to-left prompt so words like `سلام` are not typed backwards in the interactive mode. ## Common Commands Scrape the default file: ```powershell ninja-derja scrape ``` Scrape one query directly: ```powershell ninja-derja scrape "برشا" ``` Scrape another UTF-8 file: ```powershell ninja-derja scrape --file input/input.txt ``` Take only the top 3 samples per query with 8 workers: ```powershell ninja-derja scrape --top-results 3 --workers 8 ``` Write to a different output directory: ```powershell ninja-derja scrape --out output/run-01 ``` Create or show the config: ```powershell ninja-derja config init ninja-derja config show ``` Edit config from the command line: ```powershell ninja-derja config edit --top-results 5 --workers 6 --no-download-audio ``` ## Configuration Default config path: ```text ninja-derja.toml ``` Template: ```toml output_dir = "output/derja" input_path = "input/input.txt" …