rofiaayobami/datatel_communications

An end-to-end data engineering pipeline designed for DataTel Communications, a mid-sized telecom operator in Nigeria. The pipeline ingests, cleans, and consolidates millions of daily records from three disconnected operational systems (CRM, billing, Network) into a single, centralized analytics data warehouse to drive business intelligence # DataTel Communications — Data Pipeline ### AltSchool Data Engineering Capstone Project Overview An end-to-end batch data engineering pipeline for a telecommunications company that ingests customer, billing, and network session data from PostgreSQL, performs data quality validation, transforms the data into business metrics, and loads analytics-ready data into BigQuery. --- ## Business Context & Core Systems DataTel collects massive volumes of daily event telemetry across isolated production zones, each presenting distinct data quality challenges (missing values, text-formatted timestamps, and duplicate retry errors): * **Billing System (Transactional):** Captures every financial transaction, invoice, and customer payment stream. * **Network System (Event Logs):** Streams high-volume data session telemetry, including session durations and megabyte (MB) consumption. * **CRM System (Master Data):** Maintains the source-of-truth registry for customer profiles and account lifecycles. --- ## Tech Stack -Python -Apache Airflow -PostgreSQL -Google Cloud Storage (GCS) -Google BigQuery -Docker -SQL --- ## Project Structure ``` telecom_pipeline_project/ ├── dags/ │ └── datatel_pipeline_dag.py │ ├── sql/ │ ├── data_quality_checks/ │ │ ├── check_and_quarantine_billing.sql │ │ ├── check_and_quarantine_customers.sql │ │ └── check_and_quarantine_sessions.sql │ │ │ ├── staging/ │ │ ├── stg_customers.sql │ │ └── stg_network_sessions.sql │ │ │ ├── incremental/ │ │ └── incremental_billing.sql │ │ │ ├── transformation/ │ │ ├── agg_user_revenue.sql │ │ ├── agg_monthly_revenue.sql │ │ ├── agg_user_usage.sql │ │ ├── session_buckets.sql │ │ ├── agg_session_distribution.sql │ │ └── agg_arpu.sql │ │ │ └── warehouse/ │ └── dw_user_analytics.sql │ ├── docker-compose.yaml ├── requirements.txt └── README.md ``` --- ## Directed Acyclic Gr …