HillaryOnyango/Kenya-Agricultural-Regions-Weather-ETL-Pipeline
Domain:
agricultureRecord type:
software
Creator:
Hil
Host:
# Kenya-Agricultural-Regions-Weather-ETL-Pipeline
## Overview
This project implements an automated ETL (Extract, Transform, Load) pipeline using Apache Airflow to collect and store weather data for key agricultural regions in Kenya. The pipeline fetches data from OpenWeatherMap API and stores it in a PostgreSQL database.
## Features
- Daily weather data collection for 17 agricultural regions in Kenya
- Automated data pipeline using Apache Airflow
- Error handling and logging
- Environment variable configuration
- PostgreSQL database storage
- Email notifications for pipeline failures
- Email notifications for successful collection of data on a weekly basis
## Prerequisites
- Python 3.8+
- Apache Airflow 2.0+
- PostgreSQL 12+
- OpenWeatherMap API key
## Installation
1. Clone the repository:
```bash
git clone github.com
cd weather-ke-etl
```
2. Create and activate virtual environment:
```bash in Linux
python -m venv .venv
3. Install required packages:
```bash
pip install apache-airflow pandas sqlalchemy psycopg2-binary requests python-dotenv
```
4. Create `.env` file in project root:
```properties
# API Credentials
API_KEY=your_openweathermap_api_key
WEATHER_API_BASE_URL=api.openweathermap.org
# Database Configuration
POSTGRES_USER=your_username
POSTGRES_PASSWORD=your_password
POSTGRES_HOST=localhost
POSTGRES_PORT=5432
POSTGRES_DB=weather_ke
# Email Configuration (Optional)
SMTP_HOST=smtp.gmail.com
SMTP_PORT=587
SMTP_USER=hillaryonyango19@gmail.com
SMTP_PASSWORD=y**********
```
## Project Structure
```
weather-ke-etl/
├── daily_weather_etl_kenya.py # Main DAG file
├── .env # Environment variables
├── README.md # Project documentation
└── .gitignore # Git ignore file
```
## Database Schema
```sql
CREATE TABLE weather_data (
id SERIAL PRIMARY KEY,
region TEXT,
t …