vivian-okoaze/AnalystLab-Africa-Week-7-Task-Data-Pipelines-Automation

This project focused on building a simple ETL(Extract, Transform & Load) pipeline that extracts weather data from an API, transform it into a usable format and store it for analysis # Weather ETL Pipeline ## Project Overview This project builds a simple ETL (Extract, Transform, Load) pipeline that retrieves real-time weather data for multiple cities from the OpenWeather API, cleans and structures it with Pandas, and stores it for analysis in CSV format. A basic analysis is then performed to compare weather conditions across cities. ## Data Source - **API:** OpenWeather Current Weather Data API - **Cities collected:** Lagos, Kano, Port Harcourt, Abuja, Enugu, London - **Fields retrieved:** city, country, temperature, feels-like temperature, humidity, weather condition, weather description, wind speed, timestamp. ## ⚙️ How the Pipeline Works ### 1. Extract The script queries the OpenWeather Map API using a modular function `extract_weather_data(cities)` to pull real-time weather metrics. It features: * Dynamic URL parameterization. * Connection timeout safeguards (`timeout=10`). * Dynamic status code monitoring (warns and skips failed requests instead of crashing). ### 2. Transform Raw JSON payloads are parsed and cleaned within `transform_weather_data(raw_records)`: * **Key-Value Extraction:** Isolates target metrics (Temperature, Feels Like, Humidity, Wind Speed, Weather Conditions). * **Type Enforcement:** Automatically converts columns to numeric values (`pd.to_numeric`). * **Text Normalization:** Strips trailing whitespaces and enforces proper title casing on string elements (`.str.strip().str.title()`). * **Missing Value Handling:** Drops incomplete records dynamically (`.dropna()`). ### 3. Load The structured Pandas DataFrame is loaded using `load_to_csv(df, filepath)` to generate a flat CSV file ready for database injection or visualization tools like Power BI / Tableau. --- ## 📊 Live Sample Output The pipeline successfully monitors: **Lagos, Abuja, Kano, Port Harcourt, and Enugu**. ### Cleaned Dataset Preview: | City | Country | Temperature (°C) | Feels Like (°C) | Humidity (%) | Weather Condition | Description | Wind Speed (m/ …