Ruhama-Beshada/Medical-telegram-warehouse-week8

# Ethiopian Medical Telegram Data Pipeline This project implements a complete, production-style data pipeline for analyzing Ethiopian medical Telegram channels. The pipeline scrapes messages and images, transforms raw data into a structured data warehouse, enriches images using computer vision, exposes analytics through a REST API, and orchestrates the entire workflow using Dagster. ## What This Project Does The pipeline performs five tightly integrated tasks: • Scrapes messages and images from public Ethiopian medical Telegram channels • Stores raw data in a structured data lake (JSON + images) • Transforms raw data into a clean PostgreSQL data warehouse using dbt • Enriches image data using YOLO object detection • Exposes analytical insights via a FastAPI service • Automates everything using Dagster orchestration This ensures the data is reliable, testable, observable, and production-ready. ## Data Sources Public Telegram channels related to Ethiopian medical businesses, including: - Chemed - Lobelia Cosmetics - Tikvah Pharma - Additional channels from et.tgstat.com ## Pipeline Architecture Telegram → Raw Data Lake → PostgreSQL (Raw) → dbt Staging → dbt Star Schema → YOLO Image Enrichment → Analytical API (FastAPI) → Orchestrated with Dagster ## Repository Structure api/ FastAPI analytical API data/raw/ Raw JSON files and images logs/ Scraping and pipeline logs models/ dbt staging and mart models src/ Scraper, loaders, and YOLO scripts tests/ Custom dbt data tests pipeline.py Dagster pipeline definition ## Task Summary ### Task 1 – Data Scraping & Collection - Extract message ID, date, text, views, forwards, and media info - Download images per channel and message - Store raw data as JSON partitioned by date and channel - Log scraping activity and errors ### Task 2 – Data Modeling & Transformation - Load raw JSON into PostgreSQL (`raw.telegram_messages`) - Clean and standardize data using dbt staging models - Build a star schema: - …