Danielmituku/medical-telegram-warehouse

An end-to-end data pipeline for Ethiopian medical business insights from Telegram channels. Uses dbt for transformation, Dagster for orchestration, YOLOv8 for image enrichment, and FastAPI for the analytical API. # Medical Telegram Warehouse > An end-to-end data pipeline for Ethiopian medical business insights from Telegram channels. ## 📋 Overview This project builds a robust data platform that generates actionable insights about Ethiopian medical businesses using data scraped from public Telegram channels. It implements a modern ELT (Extract, Load, Transform) framework with: - **Data Extraction**: Telegram scraping using Telethon - **Data Lake**: Raw JSON storage with partitioned structure - **Data Warehouse**: PostgreSQL with dimensional modeling (Star Schema) - **Transformation**: dbt for data cleaning and modeling - **Enrichment**: YOLOv8 for image object detection - **API**: FastAPI for analytical endpoints - **Orchestration**: Dagster for pipeline automation ## 🏗️ Architecture ``` ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ Telegram │ │ Data Lake │ │ PostgreSQL │ │ Channels │────▶│ (Raw JSON) │────▶│ Warehouse │ └─────────────────┘ └─────────────────┘ └────────┬────────┘ │ ┌─────────────────┐ │ │ YOLOv8 │ │ │ (Images) │──────────────┤ └─────────────────┘ │ ▼ ┌─────────────────┐ ┌─────────────────┐ │ FastAPI │◀────│ dbt │ │ (Analytics) │ │ (Transform) │ └─────────────────┘ └─────────────────┘ ``` ## 📁 Project Structure ``` medical-telegram-warehouse/ ├── .github/workflows/ # CI/CD pipelines ├── api/ # FastAPI application │ ├── main.py # API endpoints │ ├── database.py # DB connection │ └── schemas.py # Pydantic models ├── data/ │ └── raw/ # Data lake │ ├── telegram_messages/ # JSON files (YYYY-MM-DD/channel.json) │ └── images/ # Downloaded images ├── medical_warehouse/ # dbt project │ ├── models/ │ │ ├── staging/ # Cleaned raw data │ │ └── marts/ # Star schema (facts …