wondifraw/ethiopian_medical_data_pipeline

ethiopian_medical_data_pipeline # Ethiopian Medical Data Pipeline ## 📌 Overview The **Ethiopian Medical Data Pipeline** is a modular, end-to-end data engineering platform for collecting, processing, and analyzing health-related content from Telegram channels in Ethiopia. It supports real-time public health surveillance, research data collection, and automated analytics workflows. Built with modern open-source tools—**FastAPI**, **dbt**, **Dagster**, **Docker**, and more—the pipeline offers scalable and reproducible workflows for medical data projects. --- ## 🎯 Key Goals - ✅ **Public Health Surveillance:** Monitor disease trends and misinformation. - ✅ **Automation:** Eliminate manual data collection. - ✅ **Data Quality:** Ensure structured, clean, and queryable datasets. - ✅ **Research Support:** Enable downstream analysis and ML training. - ✅ **Extensibility:** Easily plug into other channels or domains. --- ## 🧠 Use Cases - Detect and analyze COVID-19 trends from public groups. - Extract pharmaceutical mentions from chat messages. - Power analytics dashboards for NGOs and health agencies. - Train ML models on annotated medical messages or images. --- ## ⚙️ Tech Stack | Component | Tool/Library | |--------------|-----------------------------| | Language | Python 3.10+ | | API | FastAPI | | Pipeline | Dagster | | ETL | Pandas, SQLAlchemy | | Database | PostgreSQL | | Transformation| dbt | | Scraping | Telethon | | CV/Image Proc | OpenCV | | Containerization | Docker & Docker Compose | --- ## 🧭 Architecture ``` graph TD A[Telegram Channels] -->|Scrape| B[Scraper (Telethon)] B -->|Raw Data| C[data/raw/] C -->|Preprocess| D[Pandas & OpenCV] D -->|Load| E[PostgreSQL] E -->|Transform| F[dbt] F -->|Serve| G[FastAPI] D -->|Orchestrate| H[Dagster] ``` --- ## 📁 Project Structure ``` ethiopian_ …