NabloP/b5w7-shipping-a-data-product-challenge

Week 7 submission for the 10 Academy Shipping a Data Product Challenge. Builds a modular ELT pipeline to transform raw Telegram data into actionable insights on Ethiopian medical businesses, using Telethon-based scraping, dbt star schema modeling, YOLOv8 image enrichment, and a FastAPI-powered analytical interface. # B5W7: Shipping a Data Product — Week 7 Challenge | 10 Academy ## 🗂 Challenge Context This repository documents the submission for 10 Academy’s **B5W7: Shipping a Data Product** challenge. Kara Solutions, a leading data science firm in Ethiopia, aims to analyze Telegram channels related to Ethiopian medical businesses. This project builds a production-grade ELT pipeline that scrapes unstructured Telegram data, enriches it with computer vision, and delivers structured insights through a dimensional data warehouse and an analytical API. Key questions addressed: - What are the most frequently mentioned medical products or drugs across Telegram? - How does price or availability vary across channels? - Which channels post the most visual content? - What are the daily and weekly trends in health-related discussions? This end-to-end pipeline is built using Telethon, dbt, YOLOv8, FastAPI, and Dagster. --- ## 🛠 Project Features - 📥 **Data Ingestion**: Scraping public Telegram channels using the Telethon API - 🗃 **Data Lake**: Raw JSON and images organized in a partitioned file system - 🛠 **Dimensional Modeling**: dbt-based star schema built in PostgreSQL - 🧼 **Transformation**: Multi-layered staging and data marts with dbt tests - 🧠 **YOLOv8 Enrichment**: Detects objects in medical product images - 🌐 **FastAPI Interface**: Exposes insights via custom analytical endpoints - 📆 **Dagster Orchestration**: Schedules and monitors the full ELT pipeline --- ## 🔧 Project Setup 1. Clone the repository: ```bash git clone github.com cd b5w7-shipping-a-data-product-challenge ``` 2. Create and activate the virtual environment: **On Windows (PowerShell):** ```powershell python -m venv data-product-challenge .\data-product-challenge\Scripts\Activate ``` **On macOS/Linux:** ```bash python3 -m venv data-product-challenge source data-product-challenge/bin/activate ``` 3. Install dependencies: ```bash pip install -r req …