Miheret-Girmachew/PharmaPulse-ET

A full-stack data engineering platform for analyzing Ethiopia's pharmaceutical market using public Telegram data. Features a containerized ELT pipeline, dbt-modeled star schema, AI-powered image enrichment with YOLOv8, and a FastAPI backend for serving insights. Orchestrated using Dagster. # PharmaPulse-ET: End-to-End Telegram Data Analytics Platform An end-to-end data pipeline for the Ethiopian medical sector, leveraging public Telegram data. This project extracts raw data, processes it through a modern ELT framework with dbt and Dagster, enriches it with YOLOv8, and serves insights via a FastAPI application. ## 1. Overview In Ethiopia's digital marketplace, public Telegram channels are a primary source for information on medical products. This project, **PharmaPulse-ET**, tackles the challenge of transforming this chaotic, unstructured data into a structured, queryable, and valuable asset. It builds a robust data platform to ingest, process, enrich, and serve insights, enabling analysts to answer critical business questions like: - What are the most frequently mentioned medical products? - How does product availability vary across channels? - What are the daily and weekly trends in posting volume? ## 2. System Architecture The platform is built on a modern, layered data architecture that ensures scalability, reliability, and reproducibility. The data flows through several distinct stages, from raw ingestion in a data lake to a final, analytics-ready star schema. **The data pipeline consists of five key stages:** 1. **Extract & Load (to Data Lake):** Python scripts using **Telethon** scrape raw messages and images from public Telegram channels into a local file system, which serves as our data lake. 2. **Load (to Data Warehouse):** A loading script takes the raw JSON files and populates a `raw` schema in our **PostgreSQL** data warehouse. 3. **Transform & Model:** **dbt** connects to the warehouse, transforming the raw data into clean `staging` models and then building a final, analytics-ready **Star Schema** in a `marts` schema. 4. **Enrich:** A Python script using a pre-trained **YOLOv8** model performs object detection on scraped images and writes the results back to the warehouse. 5. **Serve & Orchestrate:** A **FastAPI** applic …