rahelFM/ethiopian-medical-data-warehouse

# Ethiopian Medical Telegram Data Pipeline This project is an end-to-end data platform that scrapes public Telegram channels related to Ethiopian medical businesses, stores the data in a structured raw format (data lake), and prepares it for transformation and enrichment in a modern ELT pipeline. The goal is to analyze product mentions, image content, pricing trends, and posting activity. --- ## 🚀 Project Objectives - Scrape messages and images from public Telegram channels. - Store raw, partitioned data in a data lake. - Load the raw data into PostgreSQL for transformation. - Prepare for YOLOv8 image enrichment and analytical API deployment using FastAPI. --- ## 📁 Project Structure ```bash . ├── data/ │ └── raw/ │ └── telegram_messages/ │ └── YYYY-MM-DD/ │ └── channel_name.json │ └── images/ │ └── channel_name/ ├── scripts/ │ ├── scrape_telegram.py # Scrapes messages and images from Telegram │ ├── load_raw_to_postgres.py # Loads JSON data into raw.telegram_messages table │ └── test.py # One-time login script for Telegram (session setup) ├── Dockerfile ├── docker-compose.yml ├── .env # Environment variables (not committed) ├── .gitignore ├── requirements.txt └── README.md ✅ Tasks Completed 🔧 Task 0: Project Setup & Environment Management ✅ Initialized Git repository ✅ Created requirements.txt with all dependencies (Telethon, dbt, FastAPI, YOLO, Dagster, etc.) ✅ Wrote Dockerfile and docker-compose.yml to containerize the app and database ✅ Configured .env file for secrets (Telegram API credentials, DB settings) ✅ Verified reproducible local setup 📥 Task 1: Data Scraping and Collection ✅ Used Telethon to scrape messages and images from public channels like: @lobelia4cosmetics @tikvahpharma ✅ Stored raw message data in: bash Copy Edit data/raw/telegram_messages/YYYY-MM-DD/channel_name.json ✅ Downloaded image content to: swift Copy Edit data/raw/tel …