Jaki77/medical-telegram-warehouse

End-to-end ELT pipeline for Ethiopian pharmaceutical Telegram channels. Scrapes data, transforms with dbt (PostgreSQL), enriches with YOLO image detection, serves insights via FastAPI, and orchestrates with Dagster. Delivers analytics on product mentions, pricing trends, and visual content patterns. # Medical Telegram Data Pipeline ## 📋 **Project Overview** An end-to-end data pipeline for analyzing Ethiopian medical businesses using Telegram data. This project scrapes data from public Telegram channels, transforms it into a structured data warehouse, enriches it with computer vision, and exposes insights through an analytical API. **Date**: 14 Jan - 20 Jan 2026 **Challenge**: 10 Academy - KAIM 8 - Week 8 ## 🎯 **Business Need** Kara Solutions, a leading data science consultancy in Ethiopia, needs to build a robust data platform that generates actionable insights about Ethiopian medical businesses using data scraped from public Telegram channels. ### Key Business Questions: - What are the top 10 most frequently mentioned medical products/drugs? - How do prices/availability vary across different channels? - Which channels have the most visual content (pills vs creams)? - What are the daily/weekly trends in posting volume for health topics? ## 🏗️ **Architecture** ```mermaid graph LR A[Telegram Channels] --> B[Scraping Pipeline] B --> C[Raw Data Lake] C --> D[PostgreSQL Load] D --> E[dbt Transformations] E --> F[Star Schema Warehouse] F --> G[YOLO Enrichment] G --> H[Analytical API] B --> I[Image Storage] I --> G J[Dagster Orchestration] --> B J --> D J --> E J --> G J --> H ``` ## 📁 **Project Structure** ```bash medical-telegram-pipeline/ ├── config/ # Configuration files ├── data/ # Data storage (gitignored) ├── src/ # Source code ├── dbt_medical/ # dbt transformation project ├── logs/ # Log files (gitignored) ├── api/ # FastAPI application ├── docker/ # Docker configurations └── docs/ # Documentation ``` ## **Quick Start** ### 1. **Prerequisites** - Python 3.8+ - Docker and Docker Compose - PostgreSQL 15+ - Telegram API credentials ### 2. **Clone and Setup** ```bash git clone github.com cd medical-telegram-pipeline # Create vi …