awhobbs/africa-gee-pipeline

Earth Engine extraction pipeline for continent-wide SSA crop yield monitoring at GADM admin-2 scale. Optimized for batch throughput (SAR 26×, optical 8× speedup vs naive baseline). # africa-gee-pipeline Earth Engine extraction pipeline for continent-wide Sub-Saharan African crop yield monitoring at the GADM admin-2 scale (~5,000 districts × 25 years × 5 variable groups). This is a public extract of the GEE-side code from a larger research pipeline. The full project (model training, validation, web map) is described at and in the Climate Change AI workshop paper Hobbs & Anttila-Hughes, NeurIPS 2025. ## What's here - `export_gee_v2/` — the production extraction pipeline: - `extract_dynamic.py` — dynamic-time-series variables (CHIRPS, ERA5-Land, Landsat/Sentinel-2 VIs, Sentinel-1 SAR, MODIS LAI/FPAR, AlphaEarth embeddings) - `extract_static.py` — static covariates (soil, terrain, ecoregion) - `extract_rcf.py` — Random Convolutional Features over AlphaEarth embeddings (vectorized) - `build_features.py` / `build_forecast_features.py` / `combine_exports.py` — local feature assembly from Drive exports - `config.py` — central configuration (tile scale, reducers, thresholds) - `upload_gadm.py` — GADM L0/L1/L2 boundary upload helpers - `run_all.sh` — end-to-end orchestration - `gee_best_practices.md` — what we learned, with measured speedups and citations to the GEE team's guidance posts. ## Optimization summary Headline speedups from the audit against Google's Best Practices guide and Welhoelter's December 2024 `reduceRegions` post: - SAR: 147 min → 5.6 min per task (~26×) - Optical (Landsat/S2 VIs): 15 min → 1.9 min (~8×) - AlphaEarth embedding: 19.7 min → 1.2 min (~16×) Single biggest win: stripping `ee.Algorithms.If` everywhere and replacing it with a `_safe_reduce` helper that merges a fully-masked dummy image to preserve band schema when collections are empty. Full story with all 18 optimization levers, anti-patterns we tested, and source quotes from Google team posts: see `gee_best_practices.md`. ## Requirements - Python 3.10+ - `earthengine-api`, `google-cloud-storage`, `pandas`, `numpy` - A GEE project (Contributor tier or higher r …