chalasimon/Telegram-Medical-Insights

A modern ELT (Extract, Load, Transform) pipeline that scrapes public Telegram channels for Ethiopian medical business data, enriches it with YOLOv8 object detection, models it in a star schema using dbt, and exposes insights via a FastAPI analytical API. Orchestrated with Dagster for reproducibility. # Telegram Medical Insights A modern data pipeline for analyzing Ethiopian medical businesses using public Telegram channel data. The project integrates scraping, ELT pipelines, dbt transformations, and enrichment with YOLO-based image object detection. This README provides an overview of the project, its features, and how to set it up and run it locally. ## Table of Contents - Project Overview - Features - Prerequisites - Installation - Project Structure - Running the Project - Environment Variables - Contributing --- ## Project Overview The project builds a **reproducible data platform** for collecting, storing, transforming, and analyzing data from Telegram channels. Key functionalities include: - Extracting messages and images from public Telegram channels - Storing raw data in a data lake (`data/raw`) - Transforming data into a **dimensional star schema** in PostgreSQL using **dbt** - Enriching data using **YOLOv8 object detection** - Exposing an analytical API via **FastAPI** - Orchestrating the pipeline with **Dagster** --- ## Features - **Telegram scraping**: Collect messages, media, and metadata - **Data lake & warehouse**: Layered structure for reliable ELT - **Data modeling**: Star schema with fact & dimension tables - **Data enrichment**: YOLOv8 object detection on images - **Analytical API**: Query insights such as top products, channel activity, and search messages - **Interactive Dashboard**: Streamlit dashboard for business insights, including: - Product search and time series visualization - Top mentioned products (bar chart, summary panel) - Channel activity with filters and trend analysis - Date range and product/channel filters for custom analysis - Business insights panels (e.g., most active channel, total messages) - **Dagster orchestration**: Automated end-to-end workflow using Dagster, including notebook execution with papermill - **Reproducible environment**: Dockerized Python and PostgreSQL setup --- ## Prerequisites - Python …