abeladamushumet/telegram_medical_pipeline

End‑to‑end medical data ETL pipeline extracts Amharic Telegram messages, preprocesses and cleans data, performs NER for medical entities, stores structured records, and builds a vendor analytics engine. # Shipping a Data Product: From Raw Telegram Data to an Analytical API ## Overview This project delivers a full ELT (Extract → Load → Transform) pipeline for analyzing public Ethiopian medical Telegram channels. The system scrapes messages and media, processes them, enriches them using object detection, and serves insights via an API—all orchestrated using Dagster. Built for Kara Solutions, this pipeline supports better understanding of medicine promotions, pricing, and trends via Telegram. --- ## Tech Stack | Layer | Tool/Library | Purpose | |----------------|------------------|-------------------------------------| | Extraction | Telethon | Telegram scraping | | Storage | PostgreSQL | Data warehouse | | Transformation | dbt | Star schema modeling, tests | | Enrichment | YOLOv8 (Ultralytics) | Image object detection | | API | FastAPI + Pydantic| Insightful, structured API | | Orchestration | Dagster | Schedule and monitor pipeline | | Deployment | Docker + .env | Reproducible, secure setup | --- ## Architecture ```text ┌──────────────┐ ┌─────────────┐ ┌────────────┐ ┌────────────┐ ┌────────────┐ | Telegram API | --> | Raw DataLake| --> | PostgreSQL | --> | dbt Models | --> | FastAPI | └──────────────┘ └─────────────┘ └────────────┘ └────────────┘ └────────────┘ ↘ YOLOv8 Enrichment ``` --- ## Features - Scrapes structured + unstructured Telegram content (text, images) - Stores raw JSON by date and channel in a data lake - Loads into PostgreSQL with schema - Cleans and transforms with dbt (star schema: facts + dimensions) - Detects objects (e.g., pills, creams) using YOLOv8 - Serves analytics via RESTful FastAPI - Fully orchestrated via Dagster with scheduling - Includes EDA notebook for visualization - Containerized with Dock …