Ethel-Star/Week-7.Ethiopian-medical-business-Data-Warehouse

As a data engineer at Kara Solutions, I am building a data warehouse for Ethiopian medical business data scraped from the web and Telegram. The project includes integrating YOLO object detection for analysis and using ETL/ELT frameworks to ensure clean, consistent data for efficient querying, analysis, and decision-making. # Ethiopian Medical Data Warehouse Project ## Overview As a data engineer at Kara Solutions, you are tasked with building a data warehouse that will store and analyze data related to Ethiopian medical businesses. This data will be scraped from web sources and Telegram channels. The warehouse must be scalable, efficient, and capable of handling data scraped from various platforms, including Telegram. You will also integrate object detection capabilities using YOLO (You Only Look Once) to enhance data collection and analysis. The main goal of the project is to create a centralized data warehouse that makes it easier to analyze trends, patterns, and correlations within the data, helping decision-makers gain valuable insights. The project will also include efficient querying and reporting capabilities to provide actionable intelligence. ## Project Breakdown The project consists of the following key tasks: 1. **Data Scraping and Collection Pipeline** 2. **Data Cleaning and Transformation** 3. **Object Detection Using YOLO** 4. **Expose Collected Data Using FastAPI** --- ### Task 1: Data Scraping and Collection Pipeline #### Telegram Scraping The first task is to scrape data from relevant Telegram channels. You will be using the Telegram API or custom Python scripts to extract data from these channels. Some of the key channels include: - DoctorsET - Chemed Telegram Channel - Yetenaweg - EAHCI - Additional channels can be found at ET TG Stat. For this, you will use Python packages such as `telethon` to automate the extraction of data from these Telegram channels. The process may involve creating custom scripts or simply exporting data directly from the Telegram application. #### Image Scraping for Object Detection Along with text data, images from selected Telegram channels will be scraped for object detection. The relevant Telegram channels include: - Chemed Telegram Channel - Yetenaweg The images will be processed …