benitmatumona/SA-Banking-Transaction-ETL-Pipeline

🇿🇦 South African Banking Data Platform | PostgreSQL, Python, SQL, Airflow & Data Warehousing | Customer Analytics, Fraud Detection, Revenue Analysis & Automated ETL Pipelines cat > README.md Update the `DB_USER`, `DB_PASSWORD`, `DB_NAME` and `DB_HOST` values in `src/config.py` to match the credentials in `docker-compose.yml` (or move them to environment variables) before running the ETL pipeline. --- # 📊 Analytics The project answers business questions such as (see `sql/analytics/`): - Customer segmentation by province — `customer_segmentation.sql` - Monthly revenue trends — `monthly_revenue.sql` - Fraud detection — `fraud_detection.sql` - Dormant accounts — `dormant_accounts.sql` - Top spending customers — `top_spenders.sql` - Provincial revenue analysis — `province_revenue.sql` --- # ✅ Testing Unit tests cover both the validation logic and the load logic (with the database mocked, so no live Postgres instance is required): ```bash uv run pytest tests/ -v ``` --- # 📚 Skills Demonstrated ## Python - File handling - Data structures - Functions - Error handling - Modular programming ## Pandas - Reading CSV files - Data transformation - Data validation - DataFrame manipulation ## SQL - Database creation - Table creation - Primary Keys - Foreign Keys - NOT NULL constraints - Indexes - Joins - Aggregations - Window functions ## PostgreSQL - Database design - Constraints - Indexing - Query optimization ## Data Engineering - ETL pipelines - Data validation - Data modeling - Data loading - Workflow automation ## DevOps - Docker / Docker Compose - Git - GitHub - Apache Airflow - Automated testing (pytest) --- # 🎯 Learning Objectives This project demonstrates the skills expected from a Junior Data Engineer while building the foundation required for Machine Learning Engineering. Topics covered include: - Relational database design - ETL development - Data quality validation - Workflow orchestration - SQL analytics - Production project organization --- # 🚧 Current Progress ## ✅ Completed - Customer data generator - Account data generator - Transaction data generator - PostgreSQL database creation - Database schema - …