Egbewattnkongho13/drill_data_analysis
Domain:
environment and energydigital infrastructureRecord type:
software
Creator:
Egb
Host:
A cloud-based data pipeline for aggregating, cleaning, and modelling operational data from regional oil and energy institutions across West Africa — built to enable ML-driven infrastructure risk detection.
# Drill Data Analysis
### Description
This project implements a serverless data processing pipeline on AWS for drill data analysis. It leverages AWS Lambda functions, Amazon S3 for data lake storage (Bronze, Silver, Gold layers), and Amazon ECR for Docker image management. The pipeline processes raw data, transforms it, and stores it in a structured format for analysis.
### Table of Content
- Description
- Architecture
- Technologies Used
- Getting Started
- Prerequisites
- Installation & Setup
- Usage
- CI/CD Pipelines
### Architecture
The following diagram illustrates the architecture of the drill data analysis pipeline:
```mermaid
graph TD
subgraph "AWS ECR (Elastic Container Registry)"
subgraph "ingestion-lambda-ecr"
DockerImageIngest["Docker Image"]
end
subgraph "silver-lambda-ecr"
DockerImageSilver["Docker Image"]
end
subgraph "gold-lambda-ecr"
DockerImageGold["Docker Image"]
end
end
subgraph "AWS Lambda Functions"
IngestionLambda("Ingestion Lambda (IL) [role: ingestion-lambda-role] [xp: Assumes oye-dl-bronze-ingestion-role]")
SilverTransformLambda("Silver Transform Lambda (STL) [role: silver-transform-lambda-role] [xp: Assumes oye-dl-silver-transform-role]")
GoldTransformLambda("Gold Transform Lambda (GTL) [role: gold-transform-lambda-role] [xp: Assumes oye-dl-gold-transform-role]")
end
subgraph "Oye Data Lake (AWS S3)"
Bronze[("Bronze Bucket")]
Silver[("Silver Bucket")]
Gold[("Gold Bucket")]
end
DockerImageIngest --> IngestionLambda
DockerImageSilver --> SilverTransformLambda
DockerImageGold --> GoldTransformLambda
IngestionLambda -->|write| Bronze
SilverTransformLambda -->|read| Bronze
SilverTransformLambda -->|write| Silver
GoldTransformLambda -->|read| Silver
GoldTransformLambda -->|write| Gold
classDef bucket fill:#FF9900,stroke:#000,stroke-width:2px,color:#000
classDef lambda fill:#527FFF,stroke:#000,stroke-width:2px,color:#fff
classDef ecr fill:#f9f9f9,stroke:#333,stroke-width:2px,color:#000
classDef image fill:#232F3E,stroke:#FF9900,strok …