NembotJules/Air-Quality-Ingestion-Pipeline

The Ingestion Pipeline for my Cameroon Air Quality Prediction Project # Air Quality Ingestion Pipeline This project implements a data ingestion pipeline that retrieves weather and air quality data for various cities in Cameroon using the OpenWeatherMap API. The collected data is transformed and saved to an Amazon S3 bucket using Prefect for workflow orchestration. ## Project Structure AIR-QUALITY-INGESTION-PIPELINE │ ├── data/ ├── pipelines/ │ └── data_ingestion.py ├── README.md ├── requirements.txt └── Dockerfile - `data/`: Directory where the combined data will be stored. - `pipelines/`: Contains the main Python script for data ingestion (`data_ingestion.py`). - `README.md`: Documentation for the project. - `requirements.txt`: Lists the Python dependencies required for the project. - `Dockerfile`: Configuration file for building the Docker image. ## Prerequisites - Docker installed on your machine. - An account with OpenWeatherMap to obtain an API key. - An AWS account to create an S3 bucket and obtain access keys. ## Setup 1. **Create an S3 Bucket:** Ensure you have an S3 bucket named `zeguild-bucket` in your AWS account. This is where the combined data will be stored. 2. **Clone the Repository:** ```bash git clone github.com cd AIR-QUALITY-INGESTION-PIPELINE 3. **Set Up Environment Variables:** You'll need to set the following environment variables: ```bash export WEATHER_API_KEY= export AWS_ACCESS_KEY_ID= export AWS_SECRET_ACCESS_KEY= 4. **Build the Docker Image:** ```bash docker build -t air-quality-ingestion-pipeline . ``` 5. **Run Docker Container** ```bash docker run -e WEATHER_API_KEY= \ -e AWS_ACCESS_KEY_ID= \ -e AWS_SECRET_ACCESS_KEY= \ -p 8080:8080 -p 4200:4200 \ air-quality-ingestion-pipeline ``` 6. **Access the Prefect UI:** - Prefect Server: localhost - Prefect UI: localhost ## How it works The pipeline performs the following steps: - Extract Weather Data: Retrieves weather data for specified cities using the Op …