ZED-abd/weather-pipeline

Ce que fait ce projet C'est un pipeline d'ingestion de données météo qui : Appelle l'API OpenWeatherMap pour 4 villes marocaines (Marrakech, Tanger, Dakhla, Essaouira) Enveloppe les données brutes avec des métadonnées d'ingestion Stocke le tout dans MinIO (S3 local) ou en fallback local dans ./bronze/ # Weather Data Ingestion — Bronze Layer Python pipeline that pulls **current weather** from the OpenWeatherMap free API for Moroccan cities and lands the raw JSON (plus lightweight ingestion metadata) into **MinIO** (S3-compatible) or a **local `./bronze` mirror** when MinIO is offline. This simulates a **data lake Bronze** landing zone: immutable, append-friendly JSON snapshots partitioned by `city` and `date`. --- ## 1. Prerequisites - **Python 3.10+** recommended (uses modern type-hint syntax). - **pip** for dependency installation. - A **network connection** for OpenWeather and (optionally) MinIO. - **bash** (Git Bash / WSL on Windows) if you want to use `scripts/download_minio.sh`. --- ## 2. Installation ```bash cd weather-pipeline python -m venv .venv # Windows (cmd) .venv\Scripts\activate # macOS / Linux source .venv/bin/activate pip install -r requirements.txt ``` --- ## 3. Download & start MinIO (no Docker) ### Download the binary From the project root: ```bash bash scripts/download_minio.sh ``` This auto-detects **Linux**, **macOS**, or **Windows (Git Bash/MSYS)** and downloads the official binary into the project directory (`./minio` or `./minio.exe`). On **native Windows** without Bash, download manually from: `dl.min.io` and place it next to `main.py` (or update `MINIO_BIN` in `scripts/start_minio.bat`). ### Start the server **Linux / macOS** ```bash chmod +x scripts/start_minio.sh # first time only ./scripts/start_minio.sh ``` **Windows** ```cmd scripts\start_minio.bat ``` By default this uses a local folder `./minio-data` for object storage. The API listens on **`localhost`** unless you override MinIO’s own environment variables. Default root credentials (change for real deployments): - Access key: `minioadmin` - Secret key: `minioadmin` --- ## 4. OpenWeather free API key 1. Create an account at OpenWeatherMap. 2. Navigate to **API keys**: openwea …