ademkechiche/tunisia-weather-pipeline

# Tunisia Weather Pipeline — End-to-End Data Engineering Project --- ## Table of Contents 1. Project Description 2. Architecture 3. Data Sources 4. Technology Stack 5. Installation 6. Dashboard Usage 7. Tests 8. Limitations & Future Improvements --- ## 1. Project Description This project implements an end-to-end data engineering pipeline for monitoring weather conditions across selected Tunisian cities. The goal is to demonstrate the full data lifecycle: - **Ingestion** — pulling data from a live REST API and a local CSV file - **Storage** — persisting raw data as JSON files before transformation - **Transformation** — cleaning and enriching data with Pandas - **Scheduling** — automated pipeline execution every 5 minutes - **Visualization** — interactive dashboard for weather insights **Use case:** provide near real-time weather insights and forecasts for different regions in Tunisia. --- ## 2. Architecture ``` Open-Meteo API ──┐ ├──► Raw JSON files ──► Pandas transform ──► SQLite ──► Streamlit dashboard data/cities.csv ─┘ ``` Pipeline flow: 1. Weather data is fetched from the Open-Meteo API and city metadata is loaded from `data/cities.csv` 2. Raw API responses are saved as JSON files for auditability 3. Pandas transforms and enriches the raw data 4. Processed records are stored in SQLite 5. The Streamlit dashboard reads from SQLite and refreshes automatically Full diagram: `docs/architecture.mmd` --- ## 3. Data Sources | Source | Type | Description | |--------|------|-------------| | Open-Meteo API | REST API | Current weather and forecast data — no API key required | | `data/cities.csv` | Local file | Tunisian cities metadata: name, region, latitude, longitude | --- ## 4. Technology Stack | Tool | Version | Purpose | |------|---------|---------| | Python | 3.11 | Core language | | Requests | latest | API data ingestion | | Pandas | latest | Data transformation and aggregation | | SQLite | built-in | Lightweight structured storage | | AP …