Moyijo99/weatherdata

An end-to-end analytics engineering pipelie that ingests hourly weather forecasts from the Open-Meteo API, transforms them using dbt, and delivers actionable precipitation risk classifications for field operations in Nigeria. Combines REST ingestion, BigQuery analytics, dbt transformation, and Looker Studio BI into a production-ready system. # Abuja Weather Intelligence Pipeline > **Business problem:** Logistics operators, agricultural planners, and field operations teams in Nigeria make daily decisions affected by weather — but there is no centralized, queryable, historically-accumulating weather store for Nigerian cities. This pipeline creates one, and surfaces a daily operational risk signal on top of it. --- ## What This Project Actually Does Every hour, this pipeline: 1. Fetches hourly forecast data for Abuja from the Open-Meteo API 2. Streams it into a raw BigQuery table exactly as received 3. Runs dbt transformations that clean, deduplicate, and roll up the data into a mart table 4. Produces a `precipitation_risk` signal per calendar day (`Low`, `Moderate`, or `High Risk`) based on precipitation thresholds — a concrete, business-interpretable output, not just raw numbers The mart table is designed to answer questions like: - *How many high-risk weather days has Abuja had this month?* - *Which days last week were safe for outdoor field operations?* - *What is the seasonal precipitation pattern across the year?* --- ## Architecture ```mermaid flowchart LR subgraph ingest [Ingestion] API[Open-Meteo API] PY[ingest.py] API --> PY end subgraph gcp [Google Cloud / BigQuery] RAW[(weather_data.abuja_hourly\nRaw — append-only, never modified)] PY -->|insert_rows_json| RAW end subgraph dbt [dbt Transformation Layer] STG[stg_weatherdata_raw\nStaging view\nTyped · Deduplicated · Null-filtered] MART[mart_daily_forecast\nMart table\nDaily aggregates · Risk bands] RAW --> STG STG --> MART end subgraph orchestration [Orchestration] AIR[Airflow / Astronomer] COSMOS[Cosmos — dbt tasks] AIR --> COSMOS end ``` ### Why this architecture? | Decision | Rationale | |---|---| | **Raw table is append-only** | Preserves source truth. If transformation logic changes, the raw data can be reprocessed without re-fetching from the API. This is a foundational production pattern. | | **Staging as a view, mart as a t …