Abenezer-Baheru/Ethiopian-Medical-Data-Warehouse

# Ethiopian Medical Businesses Data Warehouse ## Business Need Build a data warehouse to store data on Ethiopian medical businesses scraped from the web and Telegram channels. Integrate object detection using YOLO to enhance data analysis. ## Project Objectives - Develop data scraping and collection pipeline. - Develop data cleaning and transformation pipeline. - Object detection using YOLO. - Data warehouse design and implementation. - Data integration and enrichment. ## Tasks and Steps: ### 1 - Data Scraping and Collection Pipeline #### Telegram Scraping Utilize the Telegram API to extract data from channels: - DoctorsET - Chemed Telegram Channel - Lobelia4cosmetics - Yetenaweg - EAHCI - TGStat Medicine #### Image Scraping Collect images from: - Chemed Telegram Channel - Lobelia4cosmetics #### Steps 1. Use `telethon` for Telegram. 2. Store raw data temporarily. 3. Implement logging. ### 2 - Data Cleaning and Transformation #### Data Cleaning - Remove duplicates. - Handle missing values. - Standardize formats. - Validate data. - Store cleaned data. #### DBT for Data Transformation 1. **Setup DBT**: Install DBT and initialize the project. 2. **Defining Models**: Create DBT models for data transformation. 3. **Running DBT Models**: Perform the transformations and load the data into the data warehouse. 4. **Testing and Documentation**: Ensure data quality and provide context for the transformations. ### 3 - Object Detection Using YOLO #### Setting Up the Environment Ensure you have the necessary dependencies installed, including YOLO and its required libraries. #### Downloading the YOLO Model Clone the YOLO repository and install the required dependencies. #### Preparing the Data Collect images from the Chemed Telegram Channel and Lobelia4cosmetics. Use the pre-trained YOLO model to detect objects in the images. #### Processing the Detection Results Extract relevant data from the detection results, such as bounding box coordinates, confidence scores, and c …