jovinvicent10/dsai6226-food-prices-tz

Data Engineering pipeline for WFP Tanzania food market prices: Team A - DSAI 6226 # dsai6226-food-prices-tz Data Engineering pipeline for WFP Tanzania food market prices: Team A - DSAI 6226 ## Week 4 — Format Benchmark Results **Query:** Average monthly Maize price by region (admin1) **Dataset:** 61,100 rows · WFP Tanzania Food Prices | Format | Avg time (ms) | File size | vs fastest | |------------------|--------------|------------|------------| | CSV (pandas) | 121.9 ms | 7,609 KB | 1.0x | | Parquet (pyarrow)| 130.7 ms | 477 KB | 1.1x | | PostgreSQL | 118.4 ms | — | 1.0x ← fastest | **Key findings:** 1. At 61,100 rows all three formats perform equally (~120ms). Parquet's speed advantage only emerges at millions of rows. 2. Parquet delivers 15.9x compression vs CSV — same query speed at one-sixteenth the storage cost. 3. PostgreSQL wins for multi-table joins and concurrent users but requires a running server unlike file-based formats. 4. Verdict: for this dataset size, PostgreSQL is best for production queries. Parquet is best for archiving and sharing data across tools (BigQuery, Spark, Python).