BI pipeline and dashboard for Ethiopian inflation data — ESS crawler, pandera validation, Star Schema warehouse, Airflow DAG.
# Inflation and Cost-of-Living BI Dashboard (Ethiopia)
A verified, working ETL pipeline and analytical dashboard for Ethiopian Consumer Price Index (CPI) data, built on the Ethiopian Statistical Service (ESS) monthly Statistical Bulletin.
This repository replaces an earlier version of the project whose documentation had drifted from its actual implementation. Every component described below has been directly run and verified against a real ESS bulletin (Issue No. 34/2018) before being committed here. See `Capstone_Thesis_Chapters_1-4.docx` for the full written report.
## What this actually does
1. **Crawls** the public ESS uploads directory (`
ess.gov.et`) to locate new bulletin PDFs (`dags/scripts/ess_crawler.py`)
2. **Extracts** five CPI tables per bulletin using `pdfplumber` (`dags/scripts/ess_pdfplumber.py`)
3. **Validates** each extracted table against an explicit `pandera` schema (`dags/scripts/ess_schemas.py`)
4. **Loads** validated records into a PostgreSQL Star Schema warehouse via `SQLAlchemy`, with idempotent upserts (`dags/scripts/load_db.py`)
5. **Orchestrates** the above as a 5-task Apache Airflow DAG, manually triggered per ESS release (`dags/scripts/airflow_DAG.py`)
6. **Visualizes** the warehouse contents in a standalone interactive dashboard (`dashboard/index.html`)
A separate, independent connector (`dags/scripts/wb_connector.py`) pulls Ethiopia's annual CPI inflation indicator from the World Bank's public API for cross-validation context.
## What this deliberately does *not* do
- **No regional/zonal breakdown.** ESS's own bulletin reports CPI at the national level only — adding a fake regional dimension would misrepresent the source data. See `sql/schema.sql` for how a `Dim_Geography` table could be added later without restructuring anything, if ESS ever publishes disaggregated data.
- **No machine-learning forecasting.** The project is descriptive/historical, not predictive.
- **No automatic calendar-base …