KindnessofGod/webleadscraper2.0

Nigeria business lead-gen pipeline — scrapes listings, filters out businesses that already have a site, scores and tags the rest for outreach. Built to a hard ~$5 pilot budget. # Nigeria Business Lead-Gen Pipeline Scrapes local business listings from Google Maps across Nigerian cities, filters out businesses that already have a website, scores the rest for likely ability/need to buy a ₦150k-250k website, tags each by niche, and exports a deduplicated CSV list ready for outreach. Built around one hard constraint: **the pilot budget is ~$5**. Every stage is designed so a bug can't quietly burn through it -- see "Cost guardrails" below. ## How it works | Stage | What it does | File | |---|---|---| | A | Grid-search Google Places API for listings (default), or scrape the Maps web UI via Playwright + proxy | `src/places_api.py`, `src/grid.py` / `src/scraper.py`, `src/proxy.py` | | B | Free check: does the Maps-listed `website` field actually resolve? | `src/qualify_tier1.py` | | C | Cheap check: one live search-API query per remaining lead | `src/qualify_tier2.py` | | D | Score niche fit / review signal / rating into hot-warm-cold | `src/scoring.py` | | E | Normalize niche, dedupe, export CSV | `src/niche.py`, `src/dedup.py`, `scripts/export_by_niche.py` | | F | Structured JSON-lines logging, checkpointed resume, cost ledger | `src/logging_setup.py`, `src/db.py`, `src/cost_tracker.py` | Everything lives in one SQLite file (`data/leads.db` by default) -- leads, per-request logs, checkpoints, and the cost ledger. No external infra required; that's deliberate given the budget. ## EC2 setup (if starting from scratch) 1. **Launch an instance**: AWS Console -> EC2 -> Launch Instance. Ubuntu Server 24.04 LTS, `t3.small` (2GB RAM; bump to `t3.medium` if Chromium runs out of memory with several concurrent sessions), a new key pair, security group allowing SSH (22) from **your IP only**, 20GiB gp3 storage (default 8GB is tight once Chromium + logs + the DB are on it). 2. **Connect**: `chmod 400 key.pem && ssh -i key.pem ubuntu@ ` 3. **Install and clone**: ```bash sudo apt update && sudo apt upgrade -y sudo apt install -y python3 python3-venv pytho …