CatherineNditu/Maji-Ndogo-Data-Pipeline

Python data pipeline (pandas, SQLAlchemy, pytest) for ingesting and validating multi-source agricultural data, built with OOP and config-driven design. # Maji Ndogo Data Pipeline ## Overview The **Maji Ndogo Data Pipeline** is a modular ETL (Extract, Transform, Load) project that processes agricultural survey and weather datasets from the fictional nation of **Maji Ndogo**. Designed to reflect real-world agricultural systems across Africa, the project demonstrates how data engineering can improve data quality and support reliable, data-driven decision-making. The pipeline automates data ingestion, cleaning, validation, and preparation while following software engineering best practices, including object-oriented programming (OOP), modular architecture, configuration-driven development, logging, and automated testing. --- ## Features * Extracts farm survey data from a SQLite database. * Loads weather data from CSV files hosted online. * Cleans and standardizes agricultural and weather datasets. * Corrects inconsistent and misspelled categorical values. * Maps farm locations to their nearest weather stations. * Performs automated data quality validation using **Pytest**. * Validates data using statistical hypothesis testing with **SciPy**. * Produces clean, analysis-ready Pandas DataFrames for Exploratory Data Analysis (EDA). --- ## Project Structure ```text maji-ndogo-data-pipeline/ ├── data/ │ └── Maji_Ndogo_farm_survey_small.db ├── notebooks/ │ └── eda.ipynb ├── src/ │ ├── __init__.py │ ├── config_params.py │ ├── data_ingestion.py │ ├── data_field_processor.py │ └── weather_data_processor.py ├── tests/ │ └── validate_data.py ├── requirements.txt ├── .gitignore └── README.md ``` --- ## Installation ### 1. Clone the repository ```bash git clone github.com cd Maji-Ndogo-Data-Pipeline ``` ### 2. Create a virtual environment **Windows (Command Prompt)** ```bash py -m venv venv venv\Scripts\activate ``` **Windows (PowerShell)** ```powershell py -m venv venv venv\Scripts\Activate.ps1 ``` ### 3. Install dependencies ```bash pip install …