Thiyane24/InsightZone

WhatsApp-integrated data engineering pipeline for African SMEs FastAPI, PostgreSQL, Medallion Architecture, Cloudinary, ReportLab # InsightZone: Data Ingestion Pipeline and Business Intelligence Delivery Service via WhatsApp InsightZone is a production grade data engineering and backend portfolio architecture designed to automate the ingestion, transformation, and analysis of transactional data. The system extracts raw sales inputs submitted either as unstructured text or tabular files via WhatsApp, normalizes the data through a three tier pipeline, and delivers a professional analytical report in PDF format asynchronously within 60 seconds. The primary objective of this project is to demonstrate robust solutions to critical enterprise data engineering challenges: severe memory constraints in ephemeral cloud environments, perimeter security through cryptographic validation, event idempotency, request throttling, and heuristic schema alignment for inconsistent data layouts. ## Architectural Design and Data Flow The data lifecycle adheres strictly to the Medallion Architecture (Bronze, Silver, and Gold design patterns) to ensure data quality and prevent downstream analytical corruption. ``` [User Interface: WhatsApp] ---> [File/Text Payload] ---> [Meta Cloud API Webhook] | (Rate Limiter / HMAC SHA 256 Verification) v [Cloudinary Storage] <--- [ReportLab PDF Stream] <--- [FastAPI Backend Core] | (BackgroundTasks / APScheduler) v [pipeline/reader.py (Pandas)] | (Heuristic Schema Normalization) v [PostgreSQL / pipeline/metrics.py (Matplotlib)] ``` **Ingestion and Perimeter Layer:** The client submits a data payload (CSV, Excel, machine readable PDF, or raw text). The FastAPI webhook intercepts the HTTPS POST request, applies rate limiting to protect against abuse and excessive traffic, executes cryptographic signature verification, and delegates the payload to an asynchronous background worker thread. **Bronze Layer (Raw Ingestion):** The raw file or text input is ingested into memory. A dictionary-based heuristic routing mechanism detects and maps columns based on synonymous terminology acr …