Shehata-git/EgyptTempo

This project implements a complete end-to-end Big Data pipeline for spatio-temporal climate prediction over Egypt using NASA MODIS MOD11C1 land surface temperature observations from 2014–2024. processed through a MongoDB-backed ingestion pipeline and a 4-stage ETL workflow, and a custom SpatioTemporalConvLSTM neural network. # Spatio-Temporal Climate Prediction on NASA MODIS LST > Big Data Practical Task — Faculty of Computer Science & Information Systems, May 2026 A complete end-to-end Big Data pipeline that ingests 10 years of NASA MODIS Land Surface Temperature (MOD11C1) satellite data over Egypt, processes it through a MongoDB-backed ETL workflow, and trains a custom **ConvLSTM** model to forecast Egypt's spatial temperature grid one day ahead. --- ## Environment Setup This project uses `uv` for dependency management. ### Prerequisites - Python 3.13+ (declared in `.python-version`) - `uv` installed ### Install & sync dependencies ```bash # Install uv if not already present curl -Ls astral.sh | sh # Create virtual environment and install all dependencies from uv.lock uv sync ``` ### Run any script ```bash uv run python data-etl/scripts/0_earthaccess_to_mongo.py --year 2023 ``` ### Required environment variables | Variable | Purpose | |------------------|----------------------------------------------| | `EARTHDATA_TOKEN`| NASA EarthData bearer token (earthaccess auth)| | `MONGO_URI` | MongoDB connection string (default: `mongodb://localhost:27017`) | --- ## Project Structure ``` . ├── data-etl/ │ └── scripts/ │ ├── 0_earthaccess_to_mongo.py # Download HDF4 → crop Egypt → insert MongoDB │ ├── 1_fetch_and_scale.py # MongoDB → MinMax scale → raw_scaled_{year}.npy │ ├── 2_build_tensors.py # Sliding window (T=14) → X_{year}.npy, y_{year}.npy │ ├── 3_pack_and_move.py # tar.gz compress → move to Google Drive │ ├── 4_cleanup_mongo.py # Delete year's documents from MongoDB │ ├── run_pipeline.sh # Orchestrator: runs stages 0–4 for each year │ ├── consumer_dataloader.py # PyTorch Dataset w/ memory-mapped lazy loading │ └── consumer_ingestion.md # DL team guide: extract tarballs from Drive │ ├── notebo …