robegetachew/Ethiopian-Medical-Business-Data-Warehouse

# Ethiopian-Medical-Business-Data-Warehouse A comprehensive data warehouse project for scraping, cleaning, and analyzing Ethiopian medical business data from Telegram channels. This project includes data collection pipelines, data transformation using DBT, object detection with YOLO, and a FastAPI interface for data access. # **Task 1: Data Scraping and Collection Pipeline** ### **Overview** This project aims to develop a data scraping and collection pipeline to extract relevant data from public Telegram channels related to Ethiopian medical businesses. The project will utilize the Telegram API and custom scripts for data extraction, focusing on both textual data and images for object detection. ### **Channels to Scrape** The following Telegram channels will be targeted for data scraping: - Doctors Ethiopia - Chemed Telegram Channel - Lobelia Pharmacy and Cosmetics - Yetenaweg - Ethio-American Medical Trainings ### Objectives 1. **Text Data Extraction**: Scrape and collect textual data from the specified Telegram channels. 2. **Image Collection**: Gather images from the targeted channels for future object detection tasks. ## **Completed Tasks** ### 1. **Telegram Data Scraping** - Use the `telethon` library in Python to interact with the Telegram API. - Develop scripts to extract data from the specified channels. - Alternatively, export content using the Telegram application if necessary. ### 2. **Image and Data Collection** - Focus on collecting images specifically from: - Chemed Telegram Channel - Lobelia Pharmacy and Cosmetics - Ensure that the images collected are suitable for object detection tasks. ### 3. **Storing Raw Data** - **Initial Storage**: Store the scraped data in a temporary location such as a local database or flat files for further processing. ### 4. **Monitoring and Logging** - Implement logging to monitor the scraping process: - Track progress - Capture errors - Log important events during the scraping operation ## **Conclusion** Upon c …