Userhilal/RoadRisk_Morocco_upgraded_version

RoadRisk Morocco is a complete Big Data platform designed to collect, ingest, store, transform, govern and visualize road-risk related news and indicators in Morocco. # RoadRisk Morocco — Big Data Platform RoadRisk Morocco is a complete Big Data platform designed to collect, ingest, store, transform, govern and visualize road-risk related news and indicators in Morocco. The project implements a modern data architecture combining batch ingestion, real-time streaming, a Data Lake, Medallion architecture, Data Warehouse, orchestration, data quality, governance and analytical dashboards. --- ## Overview RoadRisk Morocco collects web articles related to road accidents, traffic, mobility and road safety in Morocco. The collected data is processed through a complete Big Data pipeline: ```text Web Sources / News Articles | |-- Batch ingestion with Apache Airflow |-- Streaming ingestion with Apache Kafka | v Bronze Layer - Raw articles and events | v Silver Layer - Cleaned, normalized and enriched data | v Gold Layer - Analytical datasets | v PostgreSQL Data Warehouse | v Streamlit Dashboard Governance: Apache Atlas + Data Catalog + Lineage + Quality Reports ``` --- ## Main Features - Web scraping for automatic article collection - Batch ingestion scheduled with Apache Airflow - Streaming ingestion with Apache Kafka - Data Lake storage with MinIO - Medallion architecture: Bronze, Silver and Gold layers - ETL / ELT transformations using Python - Data quality checks and quarantine zone - PostgreSQL Data Warehouse for analytical tables - Apache Atlas integration for metadata and governance - Data catalog, business glossary and lineage metadata - Streamlit dashboard for analytics and monitoring - Docker-based deployment for reproducibility --- ## Architecture Components ### 1. Data Sources The platform collects articles from web sources using Python-based web scraping. Technologies used: - Python - Requests - BeautifulSoup The scraper extracts article metadata such as title, URL, source, publication date and content. --- ### 2. Ingestion The project supports two ingestion modes. #### Batch Ingestion Batch ingestion is e …