Bettey-k/medical-telegram-warehouse

End-to-end data pipeline for scraping, transforming, enriching, and exposing analytics from Ethiopian medical Telegram channels. 🏥 Medical Telegram Analytics Pipeline An end-to-end data engineering & analytics pipeline that ingests Telegram data, transforms it into a dimensional data warehouse, enriches it with computer vision, exposes analytical insights via an API, and orchestrates the entire workflow using Dagster. 📌 Project Overview This project analyzes medical-related Telegram channels to extract insights about: Product mentions and engagement Channel activity and trends Visual content usage (images) Promotional vs non-promotional posts The pipeline follows modern data engineering best practices: Data Lake → Data Warehouse → Analytics API → Orchestration Telegram Scraper │ ▼ Data Lake (JSON files) │ ▼ PostgreSQL (raw schema) │ ▼ dbt Transformations (staging + star schema) │ ▼ YOLO Image Enrichment │ ▼ Analytics Data Mart │ ▼ FastAPI Analytical API │ ▼ Dagster Orchestration 📂 Project Structure medical-telegram-warehouse/ ├── api/ # FastAPI application │ ├── main.py │ ├── database.py │ └── schemas.py ├── data/ │ └── raw/ │ └── telegram_messages/ ├── medical_warehouse/ # dbt project │ ├── dbt_project.yml │ ├── profiles.yml │ ├── models/ │ │ ├── staging/ │ │ └── marts/ │ └── tests/ ├── src/ │ ├── scrape_telegram.py │ ├── load_raw_to_postgres.py │ └── yolo_detect.py ├── pipeline.py # Dagster pipeline ├── docker-compose.yml ├── Dockerfile ├── requirements.txt ├── .env └── README.md Task Breakdown ✅ Task 1 – Data Scraping Goal: Collect Telegram messages and media Output: JSON files stored in a data lake structure Scrapes messages and images from medical Telegram channels Stores data in date-partitioned folders Handles malformed or empty messages safely 📁 Example: data/raw/telegram_messages/2026-01-15/messages.json ask 2 – Data Modeling & Transformation (dbt) Goal: Build a trusted analytical data warehouse Raw Layer raw.telegram_messages Staging Layer stg_telegram_messages Cleaned fields Type …