wondifraw/ethiopian_medical_data_pipeline
Domain:
healthcareRecord type:
softwareproject
Creator:
won
Host:
ethiopian_medical_data_pipeline
# Ethiopian Medical Data Pipeline
## 📌 Overview
The **Ethiopian Medical Data Pipeline** is a modular, end-to-end data engineering platform for collecting, processing, and analyzing health-related content from Telegram channels in Ethiopia. It supports real-time public health surveillance, research data collection, and automated analytics workflows.
Built with modern open-source tools—**FastAPI**, **dbt**, **Dagster**, **Docker**, and more—the pipeline offers scalable and reproducible workflows for medical data projects.
---
## 🎯 Key Goals
- ✅ **Public Health Surveillance:** Monitor disease trends and misinformation.
- ✅ **Automation:** Eliminate manual data collection.
- ✅ **Data Quality:** Ensure structured, clean, and queryable datasets.
- ✅ **Research Support:** Enable downstream analysis and ML training.
- ✅ **Extensibility:** Easily plug into other channels or domains.
---
## 🧠 Use Cases
- Detect and analyze COVID-19 trends from public groups.
- Extract pharmaceutical mentions from chat messages.
- Power analytics dashboards for NGOs and health agencies.
- Train ML models on annotated medical messages or images.
---
## ⚙️ Tech Stack
| Component | Tool/Library |
|--------------|-----------------------------|
| Language | Python 3.10+ |
| API | FastAPI |
| Pipeline | Dagster |
| ETL | Pandas, SQLAlchemy |
| Database | PostgreSQL |
| Transformation| dbt |
| Scraping | Telethon |
| CV/Image Proc | OpenCV |
| Containerization | Docker & Docker Compose |
---
## 🧭 Architecture
```
graph TD
A[Telegram Channels] -->|Scrape| B[Scraper (Telethon)]
B -->|Raw Data| C[data/raw/]
C -->|Preprocess| D[Pandas & OpenCV]
D -->|Load| E[PostgreSQL]
E -->|Transform| F[dbt]
F -->|Serve| G[FastAPI]
D -->|Orchestrate| H[Dagster]
```
---
## 📁 Project Structure
```
ethiopian_ …