habteyesasfaw/ethiopian-medical-data-warehouse

# Ethiopian Medical Business Data Warehouse ## Overview This project involves building a data warehouse to store data on Ethiopian medical businesses scraped from Telegram channels. It also includes developing a data scraping and cleaning pipeline, applying object detection using YOLO on collected images, and exposing the collected data via a FastAPI-based API. ## Table of Contents - Overview - Project Structure - Setup and Installation - Task 1: Data Scraping and Collection Pipeline - Telegram Scraping - Image Scraping - Task 2: Data Cleaning and Transformation - Data Cleaning - DBT for Data Transformation - Task 3: Object Detection Using YOLO - YOLO Setup - Running Object Detection - Task 4: API Exposure with FastAPI - API Endpoints - Logging and Monitoring --- ## Setup and Installation 1. **Clone the Repository** ```bash git clone cd my_project ``` 2. **Create a Virtual Environment** ```bash python3 -m venv venv source venv/bin/activate ``` 3. **Install Dependencies** ```bash pip install -r requirements.txt ``` 4. **Install DBT (Data Build Tool)** DBT is used for data transformation tasks: ```bash pip install dbt ``` ## Task 1: Data Scraping and Collection Pipeline ### Telegram Scraping Data is scraped from public Telegram channels related to Ethiopian medical businesses. #### Steps: 1. **Install Telethon for Telegram Scraping:** ```bash pip install telethon ``` 2. **Running the Telegram Scraper:** The `telegram_scraper.py` script extracts messages, media, and links from specific Telegram channels: - Channels being scraped: - DoctorsET - Chemed Telegram Channel - Yetenaweg - EAHCI - To run the scraper: ```bash python scripts/telegram_scraper.py ``` 3. **Store Raw Data:** All raw data scraped from the channels is stored in the `data/raw/` directory in CSV or JSON format. ### Image Scraping Images are scraped from specific Telegram channels to be used for object detection. #### Steps: 1. **Running the Image Scraper:** The `image_scraper.py` script co …