Egbewattnkongho13/drill_data_analysis

A cloud-based data pipeline for aggregating, cleaning, and modelling operational data from regional oil and energy institutions across West Africa — built to enable ML-driven infrastructure risk detection. # Drill Data Analysis ### Description This project implements a serverless data processing pipeline on AWS for drill data analysis. It leverages AWS Lambda functions, Amazon S3 for data lake storage (Bronze, Silver, Gold layers), and Amazon ECR for Docker image management. The pipeline processes raw data, transforms it, and stores it in a structured format for analysis. ### Table of Content - Description - Architecture - Technologies Used - Getting Started - Prerequisites - Installation & Setup - Usage - CI/CD Pipelines ### Architecture The following diagram illustrates the architecture of the drill data analysis pipeline: ```mermaid graph TD subgraph "AWS ECR (Elastic Container Registry)" subgraph "ingestion-lambda-ecr" DockerImageIngest["Docker Image"] end subgraph "silver-lambda-ecr" DockerImageSilver["Docker Image"] end subgraph "gold-lambda-ecr" DockerImageGold["Docker Image"] end end subgraph "AWS Lambda Functions" IngestionLambda("Ingestion Lambda (IL) [role: ingestion-lambda-role] [xp: Assumes oye-dl-bronze-ingestion-role]") SilverTransformLambda("Silver Transform Lambda (STL) [role: silver-transform-lambda-role] [xp: Assumes oye-dl-silver-transform-role]") GoldTransformLambda("Gold Transform Lambda (GTL) [role: gold-transform-lambda-role] [xp: Assumes oye-dl-gold-transform-role]") end subgraph "Oye Data Lake (AWS S3)" Bronze[("Bronze Bucket")] Silver[("Silver Bucket")] Gold[("Gold Bucket")] end DockerImageIngest --> IngestionLambda DockerImageSilver --> SilverTransformLambda DockerImageGold --> GoldTransformLambda IngestionLambda -->|write| Bronze SilverTransformLambda -->|read| Bronze SilverTransformLambda -->|write| Silver GoldTransformLambda -->|read| Silver GoldTransformLambda -->|write| Gold classDef bucket fill:#FF9900,stroke:#000,stroke-width:2px,color:#000 classDef lambda fill:#527FFF,stroke:#000,stroke-width:2px,color:#fff classDef ecr fill:#f9f9f9,stroke:#333,stroke-width:2px,color:#000 classDef image fill:#232F3E,stroke:#FF9900,strok …