Priyanka-Sahoo147/Data-Engineering-Pipeline

Repository for the smart rental and recommendation system for listings in Ghana project # Smart Rental Pricing and Recommendation System for Accra ## Project Overview This project builds a data pipeline and API for real estate listings in Accra, with components for: - Web scraping real estate listings (running outside containers) - Cleaning and processing the data (running in Airflow) - Publishing data via API and S3 (running in Airflow) - Providing pricing recommendations ## Project Architecture This project follows a decoupled architecture where: 1. Web scraping happens outside containers on the host machine 2. Data is saved to S3 for durability 3. Airflow orchestrates the processing pipeline 4. FastAPI serves the processed data ``` ┌───────────────────┐ ┌───────────────────┐ ┌───────────────────┐ │ │ │ │ │ │ │ External Scraper │ ─────►│ S3 │ ─────►│ Airflow Pipeline │ │ (Host Machine) │ │ (Data Storage) │ │ (In Container) │ │ │ │ │ │ │ └───────────────────┘ └───────────────────┘ └─────────┬─────────┘ │ ▼ ┌───────────────────┐ ┌───────────────────┐ │ │ │ │ │ End Users │ ◄─────────────────────────────── │ FastAPI │ │ │ │ │ └───────────────────┘ └───────────────────┘ ``` ## Project Structure - `app/` - FastAPI application - `src/` - Core pipeline components - `scraper/` - Web scraping code - `cleaner/` - Data cleaning and processing - `publisher/` - Data publishing (API and S3) - `utils/` - Shared utilities - `airflow/` - Airflow DAGs for task orchestration - `data/` - Local data storage directories - `configs/` - Configuration files - `run_scraper.sh` - Shell script for running the scraper on the host machine ## Setup Instructions …