Alebachew424/telegram-medical-data-warehouse

An end-to-end data pipeline for scraping, transforming, enriching, and serving Ethiopian medical business data from public Telegram channels, using a modern ELT data warehouse architecture with dbt, YOLOv8, Dagster, and FastAPI. # 10 Academy: Artificial Intelligence Mastery - Week 7 Challenge ## Project: Building a Data Warehouse for Ethiopian Medical Business Data **Date:** July 09 - july 15, 2025 **Overview:** This project focuses on building a data warehouse to store and analyze data related to Ethiopian medical businesses scraped from Telegram channels. The goal is to develop a robust, scalable solution that incorporates data scraping, cleaning, transformation, object detection using YOLO, and data warehousing best practices. This project also exposes the collect data through a fast API **Business Need:** Kara Solutions, a leading data science company, requires a data warehouse to centralize and analyze data on Ethiopian medical businesses. This will enable comprehensive analysis, identification of trends, and improved decision-making for their clients. **Core Tasks:** 1. **Data Scraping and Collection Pipeline:** Extract data from relevant Telegram channels. 2. **Data Cleaning and Transformation Pipeline:** Clean and transform the scraped data for analysis. 3. **Object Detection using YOLO:** Integrate YOLO for object detection in images scraped from Telegram channels. 4. **Data Warehouse Design and Implementation:** Design and implement a data warehouse to store the processed data. 5. **Data Integration and Enrichment:** Integrate and enrich the data within the data warehouse. 6. **Expose the collect data using Fast API.** ## Deliverables: ### Task 1: Data Scraping and Collection Pipeline **Objective:** Extract data from Telegram channels and store it for further processing. **Steps:** 1. **Telegram Scraping:** - Utilize the Telegram API (e.g., via `telethon`) or custom scripts to extract data from public Telegram channels relevant to Ethiopian medical businesses. - **Example Channels:** - t.me - Chemed Telegram Channel (Link needed. Assumed to be a channel named "Chemed") - t.me - t.me - t.me …