dkkinyua/FoodPricesMonitoring

This project contains a pipeline that fetches food prices from markets around Kenya, transforms, loads the data into a PostgreSQL database and visualizes the data in Grafana Cloud. The pipeline is automated to run on a weekly basis for the API and monthly for the CSV/JSON files # Food Prices Monitoring Pipeline ## Overview This project automates the collection, transformation, storage, and visualization of food prices data from Kenyan open data sources and World Bank’s Microdata API. The pipeline is orchestrated using Apache Airflow hosted on an AWS EC2 instance, with PostgreSQL for data warehousing and Grafana for real-time visualization. ### a. Architecture The snapshot below shows the project architecture. ### b. Project File Structure ```graphql FoodPricesMonitoring/ # Root directory │ ├── airflow/ │ ├── dags/ │ └── airflow.cfg | └── airflow.db └── README.md ``` ### c. Schema Diagram The data warehouse follows a star schema, the database structure is as shown below **Fact Table:** fact_prices. Holds all PRIMARY KEYs and prices **Dimension tables:** dim_date, dim_market, dim_commodity, dim_pricetype ## Project Setup ### 1. Clone the project To clone this project, run the following commands to do so. > NOTE: For Airflow, run `airflow db init` to initialize a new instance of `airflow.cfg`, Airflow's config file and `airflow.db` a SQLite database instance. This project doesn't have these files as they have private information. ```bash git clone github.com cd FoodMonitoringPipeline ``` ### 2. Install and activate a virtual environment To install required dependencies, run the following command in your terminal > NOTE: For Windows users, run Airflow on Windows Subsystem for Linux (WSL), as Airflow doesn't natively support Windows. ```bash python3 -m venv myenv source myenv/bin/activate # MacOS/Linux myenv\Scripts\activate # Windows pip install -r requirements.txt ``` ### 3. Setup Airflow Run the following commands to setup Airflow for this project #### a. Set `AIRFLOW_HOME`. Run the following command to set `AIRFLOW_HOME` where our Airflow configuration will reside at ```bash export AIRFLOW_HOME=/path/to/FoodMonitoringPipeline/airflow ``` #### Initialize Airflow config fil …