2mdipro7/tanzania-education-pipeline

An end-to-end Python + MongoDB data pipeline with idempotent ETL, modular validation, and a live Streamlit dashboard tracking education program metrics. # 🌍 Mewaka Program Metrics - End-to-End Education Data Pipeline > **Status:** Live | Pipeline Passing | 0 Contract Failures A production-grade data engineering pipeline built for a fictional education program in Tanzania. This project simulates the full operational data stack of an NGO - from raw field data collection through to a live analytics dashboard - showcasing MongoDB data architecture, idempotent ETL design, modular validation, and real-time observability. ## 📌 Project Description This is a complete, end-to-end data engineering portfolio project built around a realistic problem: **how does a social impact organization track whether its education program is actually working — and how does it trust the data behind that question?** I designed and built a Python + MongoDB ETL pipeline that ingests synthetic field data (schools, students, attendance, assessments, facilitator visits), validates and cleans it through a modular quality layer, builds analytics-ready mart tables, and surfaces everything through an interactive Streamlit dashboard. The pipeline processes **12,246 records across 14 entity types**, runs **58 contract checks** on every execution, and catches and quarantines data quality issues automatically — with zero failures in the latest run. ## 1. The Problem Education NGOs operating at scale face a data trust problem. Field staff collect attendance and assessment records on mobile devices across dozens of rural schools. Those records get uploaded in batches — sometimes late, sometimes incomplete, sometimes with broken device IDs or missing school references. By the time the data reaches a dashboard, nobody knows: - Which records were already loaded vs. genuinely new - Whether a student's assessment score is real or a sync artifact - Which schools are generating unreliable data vs. genuine underperformance - Whether the pipeline ran cleanly this week or silently dropped records The result: program managers make decisions on data they …