befikadusata/fintech-datainfra

A data platform for Ethiopian multi-rail fintech Card processing, Mobile Money, Bank Transfer, and QR payments — built with the rigor of a real internal data platform: layered engineering decisions, live verification against a running stack, and a documented "why" behind every infra choice. # fintech-datainfra A data platform for a synthetic Ethiopian multi-rail fintech — Card processing, Mobile Money, Bank Transfer, and QR payments — built with the rigor of a real internal data platform: layered engineering decisions, live verification against a running stack, and a documented "why" behind every infra choice. The core problem this platform solves: **a merchant that accepts QR, Card, and Mobile Money simultaneously needs one unified settlement report** — did they actually receive what each rail says they should have, and where do the numbers not match? Cross-rail reconciliation is the spine everything else points at. ## At a glance - **What it is**: an end-to-end data platform — ingestion, streaming, lakehouse, warehouse, transformation, orchestration, monitoring, and IaC — built and *run*, not just diagrammed. - **The hard problem**: reconciling four independent payment rails into one settlement truth per merchant, including late-arriving events and in-place incremental updates. - **Verified, not assumed**: brought up as a live stack end-to-end, which surfaced and fixed 5 real bugs (offset tracking, publish-idempotency race, bind-mount permissions, an unimportable pinned dependency, a CI path bug) that a code-only review would have missed — see Reliability & incident findings. - **Measured, not estimated**: cost/performance claims (small-file compaction, partition pruning) are backed by reproduced, measured numbers, explicitly labeled as such when a live cloud number isn't available — see Cost & performance. - **Cloud-ready, not cloud-billed**: six Terraform modules for AWS/GCP are written, validated, and plan-tested in CI with zero cloud spend — a deliberate scope decision, see Infrastructure as code. - **Try it**: `docker compose ... --profile core up -d` and you have a running pipeline in minutes — see Quickstart. ## Tech stack | Layer | Technology | |---|---| | Languages | Python 3.12, SQL | | Ingestion / streaming | Kafka 3.8 (KR …