theoedibo/Climate-Impact-on-Crop-Availability-Automated-ETL-Pipeline
Domaine:
agricultureclimateType de record:
software
Créateur:
the
Hôte:
This project builds a fully automated ETL (Extract, Transform, Load) pipeline to analyze how weather conditions affect crop availability across selected regions in Nigeria.
# 🌾 Climate Impact on Crop Availability – Automated ETL Pipeline
## 📌 Overview
This project builds a fully automated **ETL (Extract, Transform, Load) pipeline** to analyze how weather conditions affect crop availability across selected regions in Nigeria.
The system extracts historical weather data and combines it with crop yield data to uncover patterns between **climate variables (temperature & rainfall)** and **agricultural productivity**.
---
## 🎯 Objective
To determine how weather conditions influence crop yield and availability using a reproducible, automated data pipeline.
---
## 🧠 Key Features
* ✅ Modular ETL pipeline (Extract → Transform → Load)
* ✅ Automated weekly execution (Windows Task Scheduler)
* ✅ Incremental data loading (append, not overwrite)
* ✅ Data cleaning & validation
* ✅ SQLite database storage
* ✅ Logging system for monitoring
* ✅ Exploratory data analysis & visualization
* ✅ Machine learning prediction model
* ✅ Streamlit dashboard for interactive insights
---
## 📥 Data Sources
### 🌦️ Weather Data
* Source: Open-Meteo Historical API
* Endpoint: archive-api.open-meteo.com
**Features:**
* time
* temperature
* precipitation
* region
* year
---
### 🌾 Crop Data
* Source: Synthetic dataset (generated programmatically)
**Features:**
* date
* region
* crop (Maize, Rice, Sorghum)
* yield
* year
---
## 🧱 Data Schema
### Crop Dataset
| Column | Description |
| ------ | ---------------- |
| date | Observation date |
| region | Nigerian state |
| crop | Crop type |
| yield | Daily yield |
| year | Year |
### Weather Dataset
| Column | Description |
| ------------- | ------------------- |
| time | Observation date |
| temperature | Average temperature |
| precipitation | Rainfall |
| region | Nigerian state |
| year | Year |
---
## 🔄 Pipeline Architecture
Crop Data + Weather Data
→ Extract
→ Cl …