Dawitom3000/ethiopian-medical-insights

Project import # Ethiopian Medical Insights A robust, end-to-end data pipeline and analytical API for extracting, transforming, and analyzing data from Ethiopian medical-related Telegram channels. The project leverages modern tools for scraping, data warehousing, transformation, enrichment, and analytics. ## Features - **Telegram Scraping**: Extracts messages and images from public Telegram channels. - **Data Lake**: Stores raw, unaltered data in a partitioned directory structure. - **PostgreSQL Data Warehouse**: Loads and stores raw and transformed data for analytics. - **DBT Transformations**: Cleans, models, and tests data using a star schema. - **YOLOv8 Enrichment**: Detects objects in images and links results to messages. - **FastAPI Analytical API**: Serves business insights and search endpoints. - **Dagster Orchestration**: Schedules and monitors the entire pipeline. ## Project Structure ``` ethiopian-medical-insights/ ├── data/ │ └── raw/telegram_messages/YYYY-MM-DD/channel_name/messages.json ├── src/ │ ├── main.py # FastAPI app entry point │ ├── scrape.py # Telegram scraping logic │ ├── scrape_entry.py # Scraping entry point │ ├── config.py # Environment/config management │ ├── database.py # DB connection for API │ ├── models.py # Pydantic models │ ├── schemas.py # API schemas │ ├── crud.py # Analytical queries │ └── dagster_pipeline/ # Dagster pipeline (to be implemented) ├── requirements.txt ├── Dockerfile ├── docker-compose.yml ├── README.md └── .env # Secrets (not committed) ``` ## Setup & Installation 1. **Clone the repository** 2. **Install dependencies** ```sh pip install -r requirements.txt ``` 3. **Configure environment variables** - Copy `.env.example` to `.env` and fill in your secrets (Telegram API keys, DB credentials). 4. **Run with Docker Compose** ```sh docker-compose up --build ``` ## Usage ### 1. **Scrape Telegram Data** - Run the scraper: ```sh …