Bini-fish/Maji-Ndogo-Data-Validation-Pipeline

# Maji-Ndogo Data Validation and ETL Pipeline This repository hosts the source code and documentation for the **Maji Ndogo Data Validation and ETL Pipeline**. The goal of this project is to build a robust, repeatable process to ingest, clean, and validate raw farm survey data and related weather data before it is passed to down-stream Machine Learning models. ## Project Overview The core pipeline is built around validating the integrity and quality of agricultural data, which is crucial for accurate predictions of crop yield and resource allocation. The process involves: 1. **Extraction (E):** Ingesting raw data from the SQLite database. 2. **Transformation (T):** Cleaning and processing the field and weather datasets. 3. **Validation (V):** Implementing strict data quality checks (`validate_data.py`) to ensure the data adheres to established business rules and standards. ## Getting Started ### Prerequisites * Python 3.8+ * The raw data file (`Maji_Ndogo_farm_survey_small.db`) must be placed in the `data/raw/` directory. ### Installation and Setup 1. **Clone the repository:** git clone github.com ```bash cd Maji-Ndogo-Data-Validation-Pipeline ``` 2. **Create and activate a virtual environment:** ```bash python -m venv venv source venv/bin/activate # On Linux/macOS # venv\Scripts\activate # On Windows ``` 3. **Install dependencies:** ```bash pip install -r requirements.txt ``` ## Execution The pipeline is designed to be executed by running the core scripts sequentially or via a separate orchestration file (`Testing_Pipeline.ipynb` included in the `notebooks` folder) **To run the data cleaning and validation process:** 1. Ensure all necessary configuration (e.g., database connection) is set up within the `src/` scripts. 2. Run the main ingestion and processing scripts `Testing_Pipeline.ipynb`. ## Source Code (`src/`) | File Name | Description | | :--- | :--- | | `data_ingestion.py` | Ha …