sumeyaaaa/Shipping-a-Data-Product

Build a production-style ELT pipeline that scrapes Ethiopian medical-business Telegram channels, lands the raw JSON in a data-lake folder structure, loads it into PostgreSQL, cleans and models it into a star-schema with dbt, enriches images via YOLO v8 object detection, and exposes analytical endpoints with FastAPI. # Shipping a Data Product – Week 7 Challenge ## Overview & Motivation This project builds an end-to-end **ELT pipeline** that scrapes public Ethiopian medical Telegram channels, transforms the data, enriches it with computer vision, and serves analytics via an API. We aim to answer questions like “most mentioned medical products” or “channel posting trends” by collecting raw Telegram data into a **data lake**, loading it into a **PostgreSQL** warehouse, applying **dbt** transformations (star-schema modeling), detecting objects in images with **YOLOv8**, and exposing results through **FastAPI** endpoints. This modern stack ensures data is **reproducible, testable, and ready for analysis**, aligning with the challenge goals of delivering a robust data product. ## Tech Stack - **Python:** General-purpose programming language for scripting and services. - **Docker:** Container platform for packaging applications and dependencies into isolated, portable containers. - **PostgreSQL:** Advanced open-source object-relational database for warehousing our data. - **dbt (Data Build Tool):** SQL-based transformation framework (in-warehouse ELT), used for building staging tables and a dimensional star schema. - **FastAPI:** Modern, high-performance Python web framework for building RESTful APIs. - **YOLOv8 (Ultralytics):** State-of-the-art object detection model (“You Only Look Once” v8) that is fast and accurate for image analysis. - **Dagster:** A modern data orchestrator for defining, scheduling, and monitoring complex pipelines. ## Project Structure ``` . ├── .dbt/ # dbt configuration (profiles.yml, etc.) ├── data/ │ ├── raw/ # Raw data lake (JSON outputs) │ │ ├── telegram_messages/ │ │ └── telegram_images/ │ └── clean/ # (For future cleaned data) ├── notebooks/ # Jupyter notebooks for development tasks │ ├── task_1/scraping.ipynb │ ├── task_2/load_and_dbt.ipynb │ └── task_3/yolo_enrich.ipynb ├── models/ …