sgatana/data-analytics-with-hadoop-and-spark

A practical Big Data Analytics project demonstrating Hadoop HDFS, Hadoop MapReduce (via Hadoop Streaming), Apache Spark batch processing, Spark Structured Streaming, and Spark SQL, applied to Kenya-relevant datasets: county population, agricultural production, and Nairobi traffic monitoring. # Kenya Big Data Analytics — Hadoop & Spark in the Kenyan Context A practical Big Data Analytics project demonstrating Hadoop HDFS, Hadoop MapReduce (via Hadoop Streaming), Apache Spark batch processing, Spark Structured Streaming, and Spark SQL, applied to Kenya-relevant datasets: county population, agricultural production, and Nairobi traffic monitoring. ## 1. Project overview The project has four integrated components: 1. **Hadoop HDFS + MapReduce** — total/average population per Kenyan county, from the 2019 census dataset (`hadoop/`). 2. **Apache Spark batch analytics** — cleaning, transformation, aggregation, and outlier analysis on the same population dataset (`spark/spark_population.py`, `notebooks/kenya_population_analysis.ipynb`). 3. **Spark Structured Streaming** — simulated real-time Nairobi junction traffic monitoring with congestion alerts and windowed summaries (`spark/traffic_producer.py`, `spark/spark_traffic_streaming.py`). 4. **Spark SQL** — Kenyan agricultural production analysis across four major crops (`spark/spark_agriculture_sql.py`). Every component was **actually executed** while building this project (not just written) — the numbers quoted throughout the reports and screenshots are real output captured from those runs, not hand-typed examples. ## 2. Technologies used - Python 3.11 - Apache Hadoop / HDFS concepts + Hadoop Streaming (mapper/reducer in Python) - Apache Spark 3.5.1 / PySpark (`spark.sql`, DataFrame API, Structured Streaming) - Pandas (for writing result CSVs) - Jupyter / nbconvert (notebook) - CSV and JSON data formats ## 3 & 4. Dataset sources and descriptions ### `data/county_population.csv` - **Dataset**: 2019 Kenya Population and Housing Census (KPHC), county-level totals. - **Publisher**: Kenya National Bureau of Statistics (KNBS). - **Primary verified fields**: `county_code` (official 001–047 numbering) and `population_2019` — cross-checked against two independent public sources during research for this project …