aymane70/CAN-2025-Data-Pipeline

The CAN 2025 Data Pipeline is a production-ready ELT (Extract, Load, Transform) data pipeline that processes Africa Cup of Nations tournament data. It demonstrates modern data engineering practices including data generation, cloud storage, data warehousing, and analytics-ready transformations. # 🏆 CAN 2025 Data Pipeline **A complete end-to-end data engineering pipeline for Africa Cup of Nations 2025** Features • Architecture • Quick Start • Documentation --- ## 📋 Table of Contents - Overview - Features - Architecture - Technology Stack - Project Structure - Prerequisites - Installation - Configuration - Usage - Data Model - DBT Transformations - Monitoring & Logging - Testing - Deployment - Dashboards & Analytics - Troubleshooting - Contributing - Roadmap - License --- ## 🌟 Overview The **CAN 2025 Data Pipeline** is a production-ready ELT (Extract, Load, Transform) data pipeline that processes Africa Cup of Nations tournament data. It demonstrates modern data engineering practices including data generation, cloud storage, data warehousing, and analytics-ready transformations. ### What This Pipeline Does - 🎲 **Generates** realistic fake tournament data (teams, players, matches, events, revenue) - ☁️ **Uploads** data to Google Cloud Storage for staging - 🗄️ **Loads** data into BigQuery as raw tables - 🔄 **Transforms** data using DBT into analytics-ready tables - 📊 **Enables** business intelligence and data visualization - 🔍 **Implements** data quality testing and validation --- ## ✨ Features ### Data Generation - ✅ 24 teams with realistic FIFA rankings - ✅ 552 players with positions, ages, and market values - ✅ 6 stadiums across Morocco - ✅ Complete tournament schedule (group stage + knockouts) - ✅ Match events (goals, yellow cards, red cards) - ✅ Ticket sales and revenue data ### Pipeline Features - ⚡ **Automated ETL** - One command to run the entire pipeline - 🔐 **Secure** - GCP service account authentication - 📝 **Logged** - Comprehensive logging at every step - 🧪 **Tested** - DBT tests for data quality - 📚 **Documented** - Auto-generated DBT documentation - 🔄 **Incremental** - Support for incremental loads ### Analytics Features - 📈 Team standings and rankings - ⚽ Top scorers leaderboard - 🏟️ Stadium performance metrics - 💰 Finan …