CippyCabana1109/mpesa-streaming-pipeline

# M-Pesa Streaming Pipeline ## M-Pesa Streaming Pipeline > **Real-time fraud detection pipeline for M-Pesa-like mobile money transactions** > **Kafka → PySpark → Elasticsearch** with ML-powered anomaly detection > **Kenyan fintech focus** with realistic transaction patterns and geographic data --- **Real-World Impact**: - Processes **1,000+ transactions/second** with ** 50km from Nairobi) - ML model identified 89 additional anomalies **Scenario 3: High-Frequency User Detection** - Simulated user with 20 transactions in 5 minutes - Rule-based: Flagged after 5 transactions - ML model: Flagged after 3 transactions (earlier detection) --- ## Architecture Overview ```mermaid graph TB A[Transaction Simulator] -->|JSONL| B[Kafka Producer] B -->|mpesa-transactions| C[Kafka Cluster] C -->|Stream| D[PySpark Processor] D -->|Fraud Rules| E[ML Anomaly Detection] D -->|Aggregates| F[Elasticsearch] F --> G[Kibana Dashboard] H[Spark UI] --> D I[Prometheus] --> D J[Jaeger] --> D style A fill:#e1f5fe style B fill:#f3e5f5 style C fill:#fff3e0 style D fill:#e8f5e8 style E fill:#fce4ec style F fill:#f1f8e9 style G fill:#e0f2f1 ``` **Key Features**: - **Real-time processing**: Kafka + PySpark Streaming - **ML fraud detection**: Isolation Forest with feature engineering - **Kenyan context**: Nairobi coordinates, mobile money patterns - **Visualization**: Kibana dashboards for fraud analytics - **Containerized**: Docker + docker-compose setup - **CI/CD**: GitHub Actions with testing and deployment - **Monitoring**: Prometheus + Jaeger tracing --- ## Project Overview Real-time streaming pipeline for simulated M-Pesa transactions with fraud detection capabilities. This project processes mobile money transactions in real-time, analyzes patterns, and identifies potentially fraudulent activities using machine learning. ## Tech Stack - **Apache Kafka**: Distributed streaming platform for real-time data ingestion - **PySpark Streaming**: Real-time data processing and …