samuel-nigussie/ethiopian-market-price-intelligence

# 🇪🇹 Ethiopian Market Price Intelligence System A Big Data platform for collecting, streaming, storing, processing, and visualizing Ethiopian commodity market-price data. The project combines **Apache Kafka**, **Hadoop HDFS**, **Apache Spark/PySpark**, and **Streamlit** to demonstrate an end-to-end Big Data pipeline. ## 🎯 Objectives * 📊 Analyze historical Ethiopian commodity prices. * ⚡ Simulate real-time price streaming with Kafka. * 💾 Store streaming data in HDFS. * 🔥 Process historical data using Spark. * 📈 Visualize market-price trends through an interactive dashboard. --- ## 🏗️ Architecture ### Streaming Pipeline ```text Historical Data │ ▼ Replay Producer │ ▼ Apache Kafka │ ├──────────────► HDFS Consumer ──► HDFS │ └──────────────► Analytics Consumer │ ▼ Streamlit Dashboard ``` ### Batch Pipeline ```text Historical Data │ ▼ HDFS │ ▼ Apache Spark │ ▼ Historical Analytics ``` --- ## 🧰 Tech Stack * **Python** — Programming language * **Apache Kafka** — Real-time streaming * **Hadoop HDFS** — Distributed storage * **Apache Spark / PySpark** — Big Data processing * **Pandas** — Data processing * **Streamlit** — Dashboard * **Plotly** — Visualization * **Jupyter Notebook** — Data exploration --- ## 📁 Project Structure ```text ethiopian-market-price-intelligence/ │ ├── consumers/ │ ├── analytics_dashboard.py │ └── hdfs_writer.py ├── dashboard/ │ └── app.py ├── producer/ │ └── replay_producer.py ├── spark_jobs/ │ └── analyze_prices.py ├── notebooks/ │ └── ethiopia_prices_2007_2026.ipynb ├── data/ ├── output/ ├── requirements.txt └── README.md ``` --- ## 🚀 Quick Start ### 1. Clone the Repository ```bash git clone github.com cd ethiopian-market-price-intelligence ``` ### 2. Install Dependencies ```bash python3 -m venv venv source venv/bin/activate pip install -r requirem …