lhiwi/telegram-data-product

An end-to-end data pipeline that scrapes Ethiopian medical business data from Telegram, transforms it with dbt, enriches it using YOLOv8, and exposes insights through a FastAPI-powered analytical API. Orchestrated with Dagster and containerized using Docker. # Shipping a Data Product: From Raw Telegram Data to an Analytical API This project implements an end-to-end data pipeline to analyze public data from Ethiopian medical business Telegram channels. It includes scraping data, transforming and modeling it with dbt, enriching it using object detection (YOLOv8), and exposing insights via an analytical API built with FastAPI. Orchestration is handled using Dagster, and the entire stack is containerized using Docker. --- ## Features * **Telegram Scraping**: Collects messages and media using Telethon. * **Data Lake & Transformation**: Raw JSON messages are stored and transformed into a star schema with dbt. * **YOLO Enrichment**: Uses YOLOv8 to detect objects in image data. * **Analytical API**: FastAPI exposes insights via clean RESTful endpoints. * **Orchestration with Dagster**: Jobs are orchestrated and monitored from a local UI. * **Reproducibility with Docker**: Runs the entire project using containers. --- ## Project Structure ``` telegram-data-product/ ├── .github/workflows/ci.yml # GitHub Actions CI pipeline ├── data/ │ └── raw/telegram_messages/ # Raw scraped Telegram data ├── notebooks/ # Exploration and analysis notebooks ├── src/ # Source code modules ├── scripts/ # Scraper, loader, enrichment scripts ├── tests/ # Unit and environment tests ├── dbt/ # dbt models and config ├── api/ # FastAPI app (main.py, models, schemas, etc.) ├── dagster_pipeline/ # Dagster jobs and ops ├── requirements.txt # Python dependencies ├── Dockerfile # Base Docker image ├── docker-compose.yml # Docker orchestration ├── .env # Secrets (excluded from Git) ├── .gitignore └── README.md ``` --- ## Setup & Run Locally (venv) ```bash # Clone project and ente …