ngatungaVictor/Tanzania-economic-indicators

# Tanzania Economic Indicators — Automated ETL Pipeline An end-to-end, scheduled ETL pipeline that pulls economic and development indicators for Tanzania from the World Bank Open Data API, cleans and transforms them in Python, loads them into SQLite, and hands off to R for statistical analysis and an auto-generated report. Built to demonstrate a full automation pipeline — not just a one-off script — including scheduled execution via GitHub Actions. ## Architecture ``` World Bank API │ ▼ ┌─────────────┐ ┌───────────────┐ ┌────────────┐ │ extract.py │ ──▶ │ transform.py │ ──▶ │ load.py │ │ (fetch + │ │ (clean, tidy, │ │ (SQLite + │ │ retries) │ │ derive YoY%) │ │ CSV) │ └─────────────┘ └───────────────┘ └────────────┘ │ ▼ data/tanzania_indicators.db │ ┌────────────┴────────────┐ ▼ ▼ r_analysis/analyze.R + dashboard/app.py report.Rmd (Streamlit + Plotly, (summary stats, trend interactive filters, regressions, correlations, KPIs, live charts) auto-rendered report) ``` Orchestrated by `src/pipeline.py`, and scheduled weekly via `.github/workflows/etl_schedule.yml`. ## Why this stack - **Python for extraction/transformation**: robust HTTP handling, pandas for tidy-data manipulation, easy to schedule and test. - **SQLite as the handoff layer**: no server to manage, a single portable file, and both Python and R can query it natively. - **R for analysis/reporting**: R Markdown makes "data changes → report updates automatically" trivial, and it's a natural fit for the statistical side (trend regressions, correlation analysis). - **GitHub Actions for scheduling**: free, version-controlled, and the commit history of `data/` becomes a visible audit trail of every run. - **Streamlit + Plotly for the dashboard**: reads the same SQLite file the pipeline writes, so there's no separate data-prep step — refresh the pipeline, refresh the page, the dashboard updates. …