Sirad12/senegal-weather-intelligence

From raw weather data to real-time insights — a production-grade Big Data pipeline built for Senegal 🇸🇳 · Kafka · MinIO · Streamlit · Docker # 🌍 Sénégal Environmental Monitoring Pipeline > Pipeline Big Data temps réel de monitoring environnemental au Sénégal. > Scraping météo → Kafka (3 brokers) → MinIO (Parquet) → Dashboard Streamlit. --- ## 🏗️ Architecture ``` Open-Meteo API ─┐ ├──► producer.py ──► Kafka (3 brokers) ──► store_to_minio.py ──► MinIO (Parquet) OpenWeatherMap ─┘ └──► app.py (Dashboard Streamlit) ↑ JupyterLab (analyse libre) ``` --- ## ⚙️ Services | Service | Port | Description | |---------|------|-------------| | Kafka broker 1 | 9092 | Broker principal | | Kafka broker 2 | 9093 | Broker réplique | | Kafka broker 3 | 9094 | Broker réplique | | MinIO API | 9000 | Stockage objet S3-compatible (Parquet) | | MinIO Console | 9001 | Interface web MinIO | | Dashboard | 8501 | Streamlit — visualisation temps réel | | JupyterLab | 8888 | Notebooks d'analyse | --- ## 🚀 Démarrage rapide ```bash # 1. Cloner le repo git clone github.com cd senegal-weather-intelligence # 2. Configurer les variables d'environnement cp .env.example .env # 3. Lancer tous les services docker compose up --build -d # 4. Attendre ~30s que Kafka soit healthy docker compose ps ``` ### Accéder aux interfaces | Interface | URL | Identifiants | |-----------|-----|--------------| | 📊 Dashboard Streamlit | localhost | — | | 🗄️ MinIO Console | localhost | minioadmin / minioadmin | | 📓 JupyterLab | localhost | token: dakar2024 | --- ## 📁 Structure du projet ``` senegal-weather-intelligence/ ├── .env # Variables sensibles ├── .gitignore ├── docker-compose.yml ├── producer/ │ ├── producer.py # Scraping Open-Meteo + OWM → Kafka │ ├── store_to_minio.py # Kafka → MinIO (format Parquet) │ ├── Dockerfile.producer │ └── Dockerfile.worker ├── consumer/ │ ├── app.py # Dashboard Streamlit temps réel │ └── Dockerfile.dashboard └── …