BretaOsodo/Open-LakeHouse-For-East-African-Agricultural-Market

# Open LakeHouse for East Africa Agricultural Market Data A cloud-native, end-to-end agricultural **Lakehouse Data Platform** built on AWS that ingests, validates, transforms, and aggregates agricultural data across East Africa using a **Medallion Architecture (Bronze → Silver → Gold)**. The project simulates real-world agricultural datasets including: - Weather Data - Crop Growth Data - Farmer Profiles - Field & Soil Data The platform transforms raw agricultural datasets into **analytics-ready business datasets** for agricultural intelligence, climate-smart farming, and decision-making. --- # Table of Contents - Project Overview - Architecture - Key Features - Tech Stack - Project Structure - Data Flow - Bronze Layer - Silver Layer - Gold Layer - Data Quality Validation - AWS Services Used - Step Function Orchestration - Data Model - Partition Strategy - How to Run the Project - Athena Queries - Future Improvements - Resume Highlights - Author --- # Project Overview This project simulates a **production-grade agricultural data platform** for East African agricultural intelligence. The platform: 1. Generates realistic agricultural datasets 2. Stores raw data in an AWS Bronze layer 3. Cleans and transforms data into a Silver layer 4. Performs automated data quality checks 5. Produces business-ready aggregations in a Gold layer 6. Makes datasets queryable using Amazon Athena 7. Sends failure notifications using SNS alerts The architecture follows the **Lakehouse + Medallion Architecture Pattern** commonly used in modern enterprise data platforms. --- # Architecture ## High-Level Architecture ```text Synthetic Agricultural Data │ ▼ Python Data Generator │ ▼ Amazon S3 Bronze (Raw JSON Data) │ ▼ AWS Glue Crawler │ ▼ AWS Glue ETL Jobs Bronze → Silver │ ▼ AWS Lambda Checks (Data Validation) │ ┌─────┴────────┐ │ │ Validation Pass Validation Fail │ │ ▼ ▼ Silver → Gold Amazon SNS Alert Aggregation │ ▼ Amazon S3 G …