davidamoah1/Data-Integration-and-Analytics

A unified platform that extracts, validates, transforms, integrates, analyzes, and visualizes data from multiple sectors to support decision-making across governments, enterprises, NGOs, and research institutions in Africa. # DataFlow — Africa's Data Intelligence Platform (AEDIP) v2.0.0 > **Built in Africa, for the world.** DataFlow is a production-ready ETL, analytics, and AI platform that ingests any tabular dataset, automatically maps business semantics, and generates governed KPIs and dashboards. It ships with a FastAPI REST layer, Streamlit dashboard, AI Copilot, job scheduling, audit logging, multi-tenant IAM, Africa Intelligence Layer, and performance infrastructure for millions of records. ## Why DataFlow? - **Upload any spreadsheet → get insights in minutes** — no data engineer required - **Understands African data** — currencies (GHS, NGN, KES, ZAR), regions, industries - **AI-powered** — semantic column mapping, anomaly detection, forecasting, natural language queries - **Enterprise-grade** — RBAC, audit logs, multi-tenant, Argon2, JWT, rate limiting - **Scales to millions** — Redis caching, background workers, chunked queries, connection pooling - **1,150+ tests passing** — production-validated with comprehensive CI/CD --- ## Project Structure ``` etl_project/ ├── config.py # Central configuration with .env support ├── pyproject.toml # Linting (ruff) + formatting (black) config ├── requirements.txt # Pinned Python dependencies ├── Dockerfile # Container deployment ├── docker-compose.yml # Dev: API + Dashboard + MySQL + Redis + Worker ├── docker-compose.prod.yml # Prod: + Nginx + SSL + Health checks ├── .env.example # Template for environment variables ├── .github/workflows/ci.yml # GitHub Actions CI pipeline ├── static/index.html # Beautiful landing page ├── alembic/ # Database migrations ├── dataset/ # Sample datasets ├── demo_datasets/ # 12 industry demo datasets (CSV) ├── data/ # Raw + processed data ├── etl/ # ETL engine (extract, transform, l …