Letsapatiiso07/aws-weather-pipeline

Real-time weather data pipeline using AWS services - automated ETL for South African cities # 🌤️ AWS Real-Time Weather Analytics Pipeline **Live automated weather data collection and processing system for South African cities** ## 🏗️ Architecture Overview ``` ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ EventBridge │───▶│ AWS Lambda │───▶│ Amazon S3 │ │ (Every 15min) │ │ Data Ingestion │ │ Raw Storage │ └─────────────────┘ └─────────────────┘ └─────────────────┘ │ ▼ ┌─────────────────┐ ┌─────────────────┐ │ DynamoDB │ │ CloudWatch │ │ Structured DB │ │ Monitoring │ └─────────────────┘ └─────────────────┘ ``` ## 📊 Real-Time Data Collection **Cities Monitored**: Pretoria, Cape Town, Johannesburg, Durban **Collection Frequency**: Every 15 minutes (96 data points/day per city) **Data Metrics**: Temperature, Humidity, Pressure, Weather Conditions, Wind Speed ## ⚡ Key Features ✅ **Fully Automated ETL Pipeline** - Zero manual intervention ✅ **Real-Time Processing** - Live data ingestion and storage ✅ **Dual Storage Strategy** - Raw JSON (S3) + Structured data (DynamoDB) ✅ **Error Handling & Logging** - Comprehensive monitoring via CloudWatch ✅ **Scalable Architecture** - Serverless, auto-scaling components ✅ **Cost Optimized** - Runs entirely within AWS Free Tier ## 🛠️ Tech Stack | Component | Technology | Purpose | |-----------|------------|---------| | **Compute** | AWS Lambda (Python 3.9) | Data processing & ETL | | **Storage** | Amazon S3 | Raw data lake | | **Database** | DynamoDB | Real-time querying | | **Scheduler** | EventBridge | Automated triggers | | **Monitoring** | CloudWatch | Logs & metrics | | **API** | OpenWeatherMap | Weather data source | ## 📁 Project Structure ``` aws-weather-pipeline/ ├── README.md ├── lambda_functions/ │ └── weather_ingestion/ │ ├── lambda_function.py │ └── requirements.txt ├── architecture/ │ ├── architecture_diagram.png │ └── aws_services.md ├── data_samples/ │ ├── sample_s3_data.json │ └── sample_ …