dataphyte-africa/anfani_scrapers

# anfani_scrapers Scrapes Nigerian public-data platforms and emits CSVs shaped for `anfani_api` bulk upload (companies → `CompaniesImport`, peoples → `PeoplesImport`). Pipeline: **procurement sites** (NOCOPO) → contractor names → **CAC/BOR** (company registry + beneficial owners) → CSV export. ## Setup ```bash npm install cp .env.example .env # optional tuning ``` ## Run ```bash npm run crawl:nocopo # NOCOPO: index pages + per-project OCDS JSONs → contractors npm run enrich:cac -- 50 # enrich N pending contractors via BOR/CAC (throttled) npm run review # list borderline matches (score 0.5–0.86) needing a decision npm run review -- promote " " # accept one → fetches its PSC owners npm run export # emit output/run-YYYY-MM-DD/{companies,contracts}.csv + peoples_pending.json npm run pipeline # all three sequentially npm run pending # queue status ``` ## How matching decides Per contractor: BOR PSC search first, authapp similarity search as fallback. Best-candidate bigram similarity vs query: - `>= MATCH_THRESHOLD` (default 0.86) → auto-accepted, PSC owners fetched - `REVIEW_FLOOR` (0.5) – threshold → parked in the review queue, nothing stored - below floor / nothing found → marked done (never reprocessed) Judge candidates by distinctive tokens, not score alone: "RABSCO LOGISTICS NIG. LTD" → "LOGISTICS NIG LTD" (0.82) is a false positive; "Cops Agro Ltd" → "COPS AGRO LIMITED" (0.67) is correct. Promote only when the unique words line up. Auto-added PSC owners become contractors themselves, so subsequent `enrich:cac` runs walk up ownership chains automatically. Env knobs (src/config.ts): `MATCH_THRESHOLD`, `REVIEW_FLOOR`, `NOCOPO_MAX_PAGES`, `CAC_DELAY_MS` (default 8s), `BOR_DELAY_MS` (6s), `OUTPUT_DIR`, `STATE_DB`. State lives in SQLite (`state/anfani.db`) — crawls resume where they stopped. Endpoint notes + quirks: docs/endpoints.md.