paulmanoni/tz-ecommerce-data-warehouse

End-to-end data warehouse over Tanzanian e-commerce reviews (Jumia TZ, Kupatana, Kariakoo): Python + PostgreSQL, Kimball star schema with SCD2, Swahili/English sentiment analysis, collaborative filtering, data-quality gates, Airflow DAGs and a Streamlit dashboard. # Tanzanian E-Commerce Reviews — Data Warehouse An end-to-end data warehouse over product reviews and ratings from Tanzanian e-commerce platforms (**Jumia TZ**, **Kupatana**, **Kariakoo online shops**), built with **Python + PostgreSQL**. Scrapers → raw landing zone → staging → Kimball star schema → BI marts, with bilingual **Swahili/English sentiment analysis**, a **collaborative-filtering recommender**, a declarative **data-quality suite**, **Airflow DAGs**, and a **Streamlit dashboard**. It runs end-to-end with **no API keys and no network** — bundled seed data replays through the same interface the live scrapers use. --- ## Contents - Quick start - What you get - Repo structure - Architecture - The star schema - Scraping and robots.txt - Sentiment analysis - Recommendations - Data quality - Dashboard - Airflow - Testing - Command reference - Troubleshooting --- ## Quick start **Prerequisites:** PostgreSQL 13+ running locally, Python 3.11–3.13. ```bash # 1. clone / enter the project cd commerce # 2. create the database and application role # (edit PGPORT/PGSUPERUSER first if your server is not on 5432) bash scripts/bootstrap_db.sh # 3. python environment python3 -m venv .venv .venv/bin/pip install -r requirements.txt # 4. configuration cp .env.example .env # then edit if your port/password differ # 5. build everything: schema -> seed data -> ingest -> star schema -> quality -> recommender make setup make pipeline # 6. open the dashboard make dashboard # localhost ``` Expected output from `make pipeline` (about 5 seconds): ``` [1/10] OK stg_reviews 4,200 rows [2/10] OK dim_category 11 rows [3/10] OK dim_customer 961 rows [4/10] OK dim_seller 25 rows [5/10] OK dim_product 73 rows [6/10] OK fact_review 4,200 rows ... run 5: 20 passed, 0 warned, 0 failed -> PASSED pipeline finished in 4.5s -- mart quality: PASSED `` …