ttomsin/grio-crawler

The africa data ingestion engine # grio-crawler The data ingestion engine for **Grio** — Nigeria's data infrastructure. `grio-crawler` is a standalone Go service responsible for discovering, collecting, enriching, and storing everything Nigerians generate on the internet into the Grio database. Licensed under PolyForm Noncommercial 1.0.0 — free to use, study, and modify for any noncommercial purpose. Commercial use requires a separate license — see the LICENSE file for details. --- ## What Grio Is Grio is not a news aggregator. It is not a search engine. It is not a dashboard. It is a **data layer** — the foundational layer that should have existed decades ago. A continuously running system that pulls everything Nigerians generate on the internet — news, forums, research papers, legislation, court judgments, public discussions — structures it, enriches it with AI-powered analysis, and makes it queryable through a clean API. `grio-crawler` is the engine that keeps that layer alive and growing. --- ## Architecture ``` ┌─────────────────────────────────────────────────┐ │ grio-crawler │ │ │ │ Registry │ │ └── RunAll() │ │ ├── Crawler A │ │ ├── Crawler B ──► CrawlerBuffer ──► DB │ │ └── Crawler N │ │ │ │ Redis (Upstash) ←── colly visited URL store │ └─────────────────────────────────────────────────┘ ``` - **Registry** — manages all crawlers, prevents overlapping runs via mutex guard - **CrawlerBuffer** — batches records before flushing to CockroachDB - **CrawlerAnalyzer** — runs enrichment at ingest (sentiment, risk, locations, topics) - **Colly + Redis** — tracks visited article URLs persistently across restarts - **Proxy routing** — applied per-collector where needed --- ## What Gets Collected | Category …