juniorworku/ethiopian-medical-business-data-warehouse

# Ethiopian Medical Business Data Warehouse ## Overview This project involves the development of a robust data warehouse to store and analyze data from Ethiopian medical businesses. It includes data scraping from various sources including Telegram channels, data cleaning, object detection using YOLO, and API exposure using FastAPI. The goal is to facilitate comprehensive data analysis and decision-making in the healthcare sector. ## Business Need Kara Solutions aims to centralize data on Ethiopian medical businesses to enable stakeholders to derive valuable insights through advanced analytics. The data warehouse enhances operational efficiency and strategic planning by providing a unified platform for data integration and analysis. ## Project Components ### Data Scraping and Collection Pipeline - **Telegram Scraping:** Utilize Telegram API and custom scripts to extract data from relevant channels such as DoctorsET, Chemed, lobelia4cosmetics, yetenaweg, and others listed on et.tgstat.com. - **Image Scraping:** Collect images from specified Telegram channels for object detection. - **Storing Raw Data:** Store scraped data in a temporary storage location (e.g., local database) before further processing. - **Monitoring and Logging:** Implement logging to track scraping progress, errors, and monitor pipeline performance. #### Setup Instructions Data Scraping 1. **Clone the Repository:** ```bash git clone github.com cd ethiopian-medical-business-data-warehouse 2. **Install Dependencies:** ```bash pip install -r requirements.txt 3. **Telegram Scraping Setup:** -Create a Telegram API account and generate API credentials. -Update telegram_scraper.py with your API credentials and channels to scrape. 4. **Image Scraping Setup:** -Ensure Python packages like telethon are installed for Telegram image scraping. -Modify download_images.py to specify image colle …