paulmanoni/tanzania-news-data-warehouse

End-to-end data warehouse over Tanzanian news media: continuous RSS/GDELT/scraping ingestion, Kimball star schema in PostgreSQL, Swahili+English NLP, 5-minute micro-batch, spike detection and a Streamlit BI dashboard. # Tanzanian News Data Warehouse A complete, working data warehouse built on Tanzanian news media — from continuous ingestion through a Kimball star schema to a live BI dashboard. **This README is written for a data-warehousing audience, not a programming one.** It explains what every table is for, where every number comes from, and what every chart on the dashboard is showing. You do not need to read any Python to follow it, and you do not need Docker to run it. --- ## Table of contents 1. What this project does 2. Prerequisites 3. Setup — five commands 4. The architecture, layer by layer 5. The schema explained table by table 6. Where the data comes from 7. The dashboard, chart by chart 8. How to scrape more data (no Docker) 9. Data quality 10. Orchestration with Airflow 11. Command reference 12. Repository map 13. Presentation walkthrough 14. Troubleshooting --- ## 1. What this project does Tanzanian news publishers — *Mwananchi*, *The Citizen*, *HabariLEO*, *Daily News*, *ITV/IPP Media* and *Michuzi Blog* — publish hundreds of articles a day in Swahili and English. Individually those articles are just text. Collected, modelled and measured, they answer questions no single article can: - What is the country talking about **right now**, and is any story **suddenly** accelerating? - How has coverage of a topic moved **over weeks**? - Do **state-owned and privately-owned** outlets frame the same story differently? - Which **people, parties and institutions** appear together, and how has that network changed? This project builds the warehouse that answers them: ``` News sites ──► Ingestion ──► Stream ──► Staging ──► NLP ──► Star schema ──► Marts ──► Dashboard (language, sentiment, entities, topics) ``` New articles flow in continuously and are loaded to the warehouse in **micro-batches every 5 minutes**. ### What makes it a warehouse and not just a database | Warehousing concept | Where you'll find it here | |---|---| | Dimensional (Kimball) modelling | …