Hatim-BE/Tagantino-ETL_CHIRPS

A repo for extracting, processing and loading Africa CHIRPS (precipitation) data locally and in aws s3 buckets via Docker containerization / Airflow # Tagantino-ETL_CHIRPS Extract, Transform, and Load pipeline for CHIRPS (Climate Hazards Group InfraRed Precipitation with Station data) precipitation datasets, with Airflow automation support. --- ## 🌍 Overview This project is a lightweight, modular ETL pipeline designed for acquiring and processing satellite-based rainfall data (CHIRPS). It is especially tailored for regional climate analysis, particularly drought monitoring in Morocco or other specified areas. Main features: - 📥 Download CHIRPS rainfall data (daily/monthly) - 🗂️ Automatically organize and preprocess raster data - 🧮 Process and clip raster files to geospatial boundaries - 🔄 Convert to CSV format for analytics - ☁️ Upload processed data to AWS S3 - 🔄 Airflow integration for scheduled and automated workflows CHIRPS is a 30+ year quasi-global rainfall dataset (1981–present) that combines satellite imagery with in-situ data, supporting high-resolution climate research. --- ## 🚀 Installation ### Requirements - Python 3.8 or newer - GDAL-compatible raster stack (rasterio, geopandas, numpy, etc.) - AWS CLI and `boto3` (for cloud storage modules) - Apache Airflow (for workflow automation) ### Steps ```bash # Clone the repo git clone github.com cd Tagantino-ETL_CHIRPS # Create a virtual environment python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate # Install dependencies pip install -r requirements.txt # For Airflow setup with Docker docker-compose up -d ``` ### AWS CLI Installation and Setup Follow the official AWS CLI installation instructions for your operating system: #### Linux ```bash # Download the AWS CLI installer curl "awscli.amazonaws.com" -o "awscliv2.zip" unzip awscliv2.zip # Install the AWS CLI sudo ./aws/install # Verify the installation aws --version ``` #### macOS ```bash # Download the AWS CLI installer curl "awscli.amazonaws.com" -o "AWSCLIV …