helina1abebe/telegram-medical-data

A data engineering project that scrapes, processes, and transforms medical business data from Ethiopian Telegram channels into an analytical API using Python, dbt, YOLOv8, and FastAPI. # telegram-medical-data **From Raw Telegram Data to an Analytical API** This project builds a complete data pipeline that scrapes data from Ethiopian Telegram channels about medical businesses, transforms it into structured data using dbt, enriches it using YOLOv8 image detection, and exposes it via a FastAPI endpoint. Dagster is used for orchestration and scheduling. --- ## 🧩 Project Structure telegram-medical-data/ │ ├── .env # Environment variables (API keys, DB credentials) ├── .gitignore ├── requirements.txt # Python dependencies ├── Dockerfile ├── docker-compose.yml # Docker setup for local development │ ├── src/ │ ├── scraper/ # Telegram scraping logic using Telethon │ │ └── telegram_scraper.py │ └── utils/ │ └── logger.py │ └── data/ └── raw/ # Raw Telegram messages and media ## ⚙️ Tech Stack | Layer | Tool / Library | |----------------|--------------------------| | **Scraping** | Telethon | | **ETL Logic** | Python, dbt | | **Orchestration** | Dagster | | **Enrichment** | YOLOv8 (for media/image analysis) | | **Storage** | PostgreSQL | | **API** | FastAPI | | **DevOps** | Docker, docker-compose | --- ## 🚀 Getting Started ### 1. Clone the repo git clone github.com cd telegram-medical-data ### 2. Set up your virtual environment python -m venv .venv .venv\Scripts\activate # On Windows # or source .venv/bin/activate # On Mac/Linux ### 3. Install dependencies bash Copy code pip install -r requirements.txt ### 4. Create a .env file dotenv TELEGRAM_API_ID= ______________ TELEGRAM_API_HASH= ______________ POSTGRES_HOST=db POSTGRES_PORT=5432 POSTGRES_USER=postgres POSTGRES_PASSWORD=yourpassword POSTGRES_DB=telegram_db ### 5. Run the Telegram scraper python src/scraper/telegram_scraper.py 📈 Data Pipeline Overview Scrape messages and media from selected Telegram channels. Store raw data in the /data/raw/ …