Habtu32/medical-telegram-warehouse

This data platform harnesses public Telegram data on Ethiopian medical businesses to extract, process, and analyze info for actionable insights. It empowers healthcare stakeholders with decisions via modern engineering, addressing key questions like product trends and supports scalable analytics. # Medical Telegram Warehouse — Final Project README Comprehensive end-to-end ELT data product that scrapes public Telegram channels for medical and health product information, ingests raw data into a Postgres-backed data warehouse, transforms the data with dbt into a dimensional star schema, enriches images with YOLOv8 object detection, and exposes analytical endpoints via a FastAPI service. This repository contains the code, models, and orchestration used for the 10 Academy: Artificial Intelligence Mastery Week 8 challenge (14–20 Jan 2026). **Project goals** - Build a reproducible, testable ELT pipeline for Telegram-sourced data. - Provide reliable analytical models (dbt) and tests to ensure data quality. - Enrich visual data using object detection and integrate results into the warehouse. - Serve business-facing analytical endpoints answering product-mentions, channel activity, message search, and visual content statistics. ## Architecture (high level) - Extract: `src/scraper.py` (Telethon) collects messages and images into `data/raw/`. - Load: `src/loader.py` reads raw JSON and writes to Postgres (`raw_telegram_data`). - Transform: `medical_warehouse/` is a dbt project (staging -> marts) to produce `dim_` and `fct_` tables. - Enrich: `src/yolo_detect.py` runs YOLOv8 over downloaded images and emits `data/yolo_detections.csv` to be loaded via dbt into `fct_image_detections`. - Serve: `api/main.py` exposes FastAPI endpoints backed by SQL queries against the warehouse. - Orchestration: `pipeline.py` defines a Dagster job that runs the full flow (scrape -> load -> dbt -> yolo). ## What’s in this repository - `src/` — Scraper, loader, YOLO detection, and helpers - `api/` — FastAPI app, database connection, Pydantic schemas - `medical_warehouse/` — dbt project (models, tests, docs, profiles.yml) - `pipeline.py` — Dagster job definition - `data/` — Raw JSON and images; results like `yolo_detections.csv` - `requirements.txt` — Python dependencies - `yolov8n.pt` — Y …