Guyatu1627/medical-telegram-warehouse

An end-to-end data pipeline that extracts raw data from public Ethiopian medical Telegram channels, loads it into a PostgreSQL data warehouse, transforms it into a dimensional star schema using dbt, enriches it via YOLOv8 object detection, and exposes insights through a FastAPI analytical REST API—all automated using Dagster orchestration. # Medical Telegram Warehouse A robust data pipeline and analytical warehouse for collecting, processing, and analyzing medical telegram messages. Built with **FastAPI**, **dbt (data build tool)**, and **PostgreSQL**. ## Project Architecture & Structure This repository follows a clean, decoupled architecture: - **`api/`**: A FastAPI application for exposing endpoints to query message telemetry and warehouse health. - **`medical_warehouse/`**: A dbt project for structuring, cleansing, and modeling data in the PostgreSQL warehouse. - **`data/`**: Directory for storing seeds, raw data extracts, or SQLite dumps. - **`src/`**: Shared Python utilities and processing modules. - **`notebooks/`**: Jupyter notebooks for exploratory data analysis (EDA). - **`scripts/`**: One-off utilities and run scripts. - **`tests/`**: Unit and integration test suites. ``` medical-telegram-warehouse/ ├── .vscode/ # Workspace specific settings ├── .github/workflows/ # CI/CD pipelines (unit tests) ├── data/ # Data placeholder directory ├── medical_warehouse/ # dbt project configurations & models ├── src/ # Python source directory ├── api/ # FastAPI backend ├── notebooks/ # Jupyter notebooks for analysis ├── tests/ # Test suites └── scripts/ # Maintenance scripts ``` ## Getting Started ### Prerequisites Ensure you have the following installed locally: - Python 3.10+ - Docker & Docker Compose - Git ### Local Setup 1. **Clone the Repository**: ```bash git clone cd medical-telegram-warehouse ``` 2. **Configure Environment Variables**: Copy the sample settings if you want to customize them: ```bash cp .env.example .env # Or edit the existing .env file ``` 3. **Start PostgreSQL Container**: Spin up the Postgres DB using docker-compose: ```bash docker-compose up -d db ``` 4. **Install Dependencies**: Initialize a virtual environment and install requirements: ```bash python …