atlasia-ma/Chataba-Scraper

A facebook post/comments scraper for moroccan darija 🇲🇦 ## Chtaba scraper Scrape Facebook page post links, then fetch each post's text and public comments using Selenium attached to a logged‑in Chrome session via remote debugging. ### Features - **Post links extraction**: Scrolls a Facebook page and collects unique post URLs. - **Post + comments scraping**: Opens each post and extracts the main post text and visible comments. - **NDJSON output**: Writes one JSON object per line for easy downstream processing. ### Requirements - Python 3.9+ - Google Chrome (recent version) - ChromeDriver (Selenium 4 will auto-manage via Selenium Manager in most cases) - A Facebook account logged in within the Chrome instance you attach to Python dependencies are listed in `requirements.txt`: - `selenium` - `tqdm` ### Setup 1) Create and activate a virtual environment, then install deps: ```bash python3 -m venv .venv source .venv/bin/activate pip install -r requirements.txt ``` 2) Start Chrome with remote debugging enabled (macOS example): ```bash "/Applications/Google chrome.app Chrome" \ --remote-debugging-port=9222 \ --user-data-dir="/tmp/chrome-fb-scraper" \ --disable-notifications ``` - In that Chrome window, log into Facebook so pages and posts are accessible. - The code expects the debugger address at `localhost:9222`. Windows example (Command Prompt): ```bat "C:\Program Files\Google\Chrome\Application\chrome.exe" ^ --remote-debugging-port=9222 ^ --user-data-dir="%LOCALAPPDATA%\Temp\chrome-fb-scraper" ^ --disable-notifications ``` Windows example (PowerShell): ```powershell & "C:\Program Files\Google\Chrome\Application\chrome.exe" ` --remote-debugging-port=9222 ` --user-data-dir="$env:LOCALAPPDATA\Temp\chrome-fb-scraper" ` --disable-notifications ``` - The `--disable-notifications` flag suppresses site notification popups while scraping. ### Usage There are two steps: extract post links from a page, then scrape each post and its comments. #### 1) Extract post links Script: `links_extractor.py` …