Jimmy-JayJay/malawi-climate-pipeline

An automated ETL pipeline built with Python, PostgreSQL, and Streamlit. Features daily incremental updates from NASA POWER API, error handling, and an interactive dashboard for climate analysis. # Malawi Climate Data Pipeline An automated ETL (Extract, Transform, Load) system that fetches, processes, and stores historical and real-time climate data for major cities in Malawi using NASA POWER API and PostgreSQL. ## Overview This project addresses the challenge of accessing reliable, structured climate data for Malawi without the hassle of dealing with CSV files. It provides: - **Historical data**: 10+ years of daily weather observations (2014-2024) - **Automated updates**: Incremental daily/weekly data fetching - **Production database**: PostgreSQL storage with proper schema design - **Three major cities**: Blantyre, Lilongwe, and Mzuzu ## Features - Fetches data from NASA POWER API (Temperature, Precipitation, Humidity) - Automated incremental updates (fetches only missing days) - PostgreSQL database with unique constraints (prevents duplicates) - Idempotent operations (safe to run multiple times) - Comprehensive logging - Windows Task Scheduler integration - Interactive Dashboard using Streamlit - **Future versions will include more feature, be sure to check for updates** ## Tech Stack - **Python 3.12+** - **Data Engineering**: PostgreSQL, SQLAlchemy, Pandas, Requests - **Visualization**: Streamlit, Plotly - **Data Source**: NASA POWER API ## Project Structure ``` malawi-climate-pipeline/ ├── dashboard/ │ └── app.py # Streamlit Dashboard application ├── scripts/ │ ├── fetch_nasa_power.py # Initial bulk data download │ ├── load_to_db.py # Database initialization & CSV loading │ └── daily_update.py # Incremental updates ├── docs/ │ └── setup_guide.md # Detailed installation instructions ├── config.example.py # Configuration template ├── requirements.txt # Python dependencies ├── .gitignore └── README.md ``` ## Quick Start ### Prerequisites - Python 3.12+ - PostgreSQL installed and running - Internet connection ### Installation 1. **Clone the repository** ```bash gi …