chiche-ds/secure-backup-pipeline

Open-source, policy-aware backup and replication pipeline with real-time file detection, AWS S3 upload with object lock, and full observability. That helps enforce the data privacy laws of Cameroon # Secure Backup & Replication Pipeline ### Project Overview This project is the implementation of my **Master’s thesis**: **“Design and Implementation of a Secure, Policy-Aware Data Backup and Replication Solution to the Cloud.”** It provides an **open-source, automated backup system** that ensures **data security, compliance, and audit readiness** by integrating: - **Real-time file monitoring** - **Policy-as-code enforcement** using Open Policy Agent (OPA) - **Malware scanning** and automatic quarantine - **Data compression and AES-256 encryption** - **Secure replication to AWS S3 with Object Lock** - **Full observability with Prometheus, Loki, and Grafana** This solution is designed for **SMEs and public institutions in Cameroon** to comply with **Cameroon Data Protection Law 2024/017**, especially Articles **17, 19, and 22**. --- ## Features - 🔹 **Real-time detection** of new and modified files using `inotify` - 🔹 **Duplicate detection** using SHA-256 hash database - 🔹 **Policy Enforcement #1 (Pre-processing)**: Classifies files, checks size/type - 🔹 **Malware scanning with ClamAV** and **automatic quarantine** for infected files - 🔹 **Compression and AES-256 GPG encryption** for secure storage - 🔹 **Policy Enforcement #2 (Post-processing)**: Verifies encryption & destination authorization - 🔹 **Cloud replication to AWS S3** with Object Lock (immutability) - 🔹 **Audit-ready logging** with Prometheus + Loki - 🔹 **Visual dashboard** in Grafana showing: - Files processed - Files quarantined or rejected - Upload success/failure - End-to-end latency --- ## Pipeline Architecture ```text [File Detected] ↓ [Duplicate Check] → Duplicate → [Skip & Log] ↓ [Policy Check 1] → Denied → [Skip & Log] ↓ [Virus Scan] → Infected → [Quarantine & Log] ↓ [Compress + Encrypt] ↓ [Policy Check 2] → Denied → [Hold in Staging] ↓ [AWS S3 Upload + Object Lock] ↓ [Logs & Grafana Dashboard] ``` --- ## Technology Stack - **OS**: Ubuntu Server 22.04 - **Monitoring & Logging**: Prometh …