Mariem-Jaidi/satellite-timeseries-ml

End-to-end ML pipeline for satellite vegetation index cleaning and wheat phenology extraction — INRAT Tunisia --- title: Wheat Phenology Pipeline emoji: 🌾 colorFrom: green colorTo: blue sdk: streamlit sdk_version: "1.58.0" app_file: streamlit_app.py pinned: false --- # INRAT Béja Wheat Phenology Pipeline An end-to-end automated ML pipeline for cleaning satellite-derived vegetation index data and extracting phenological metrics from wheat plots in the Béja region, Tunisia. Built during an internship at **INRAT** (Institut National de la Recherche Agronomique de Tunisie), satellite remote sensing department. --- ## What This Project Does Satellite imagery of wheat fields produces time-series data for vegetation indices (NDVI, EVI, NDRE, GNDVI, SAVI) — but cloud cover regularly corrupts or eliminates readings entirely. Previously, deciding what to do with each missing value required manual judgment. This pipeline replaces that process with an automated, statistically grounded, and fully reproducible workflow. **Input:** 29 raw CSV files (one per wheat plot), 8 varieties, Béja region **Output:** Clean dataset + audit log + RF validation report + phenology metrics --- ## Pipeline Steps | Step | Module | What it does | |------|--------|--------------| | 1 | `ingestion.py` | Reads all raw CSVs, extracts variety from filename, merges into master DataFrame | | 2 | `imputation.py` | Deduplicates dates, applies three-tier decision rule (DELETE / INTERPOLATE / FLAG) based on cloud cover % | | 3 | `curve_fitting.py` | Upgrades linear interpolation with Savitzky-Golay curve fitting for INTERPOLATE rows | | 4 | `ml_validation.py` | Trains a Random Forest on trusted rows, independently predicts interpolated values as a cross-check | | 5 | `phenology.py` | Extracts SOS, POS, EOS, LOS, AUC per plot using the 20% amplitude threshold method | --- ## Key Results - **696 raw rows** across 29 plots → **667 clean rows** after pipeline - **41 missing observations** intelligently triaged: 16 interpolated, 24 flagged, 0 deleted - **Savitzky-Golay curve fitting** upgraded linear interpolat …