radical-data/amis-harvester

Fresh produce market prices from South Africa's AMIS system # AMIS Harvester AMIS Harvester is a robust, reproducible scraper that collects fresh produce market prices from South Africa's AMIS system, validates and normalises them, and exports them as analysis-ready parquet/CSV files with full provenance metadata. ## Goal & Scope **What it does:** - Scrapes market price data from the South African Department of Agriculture's AMIS system - Builds code-to-label mappings from the landing page automatically - Fetches prices for specified markets, products, and time periods - Parses HTML tables into validated, structured data - Exports to parquet/CSV with full provenance tracking - Provides market information for context **MVP Scope:** - ✅ Build code/label maps from landing page ` ` elements - ✅ Fetch prices for given (period, market, product, variety, class, size, container) - ✅ Parse into DataFrame with stable schema - ✅ Fetch market info and parse key-value pairs - ✅ CLI interface with auto-generated filenames - ✅ TTL-based caching for code maps - ✅ Comprehensive test coverage **Out of scope (MVP):** - Schedulers, dashboards, databases, cloud infrastructure - Real-time streaming or webhooks - Data visualization or analysis tools ## Quick Start ### Installation ```bash # Clone and install git clone cd amis-harvester uv venv uv pip install -e . uv pip install -e ".[dev]" # For development ``` ### Basic Usage ```bash # Scrape 90 days of potato prices from Tshwane market harvester scrape --period 91 --market PRE --product 0001 # View available code mappings harvester maps # Export code maps for inspection harvester maps --export data/code_maps.json ``` ### Bulk Scraping for Analysis ```bash # Create a sample configuration for bulk scraping harvester bulk --create-config # Run bulk scraping with configuration file harvester bulk --config-file scraping_config.json # Or use command line parameters (single period only) harvester bulk --markets PRE,CAP,JOH --products 0001,0042 --period 31 # Use 'all' for comprehensive …