habeshaethiopia/ethiomed-datahub

A scalable data warehouse for Ethiopian medical businesses, integrating web and Telegram scraping, ETL/ELT pipelines, and YOLO-based object detection. This project enables efficient data collection, transformation, and analysis to uncover insights and trends in the medical sector. # **Ethiomed Data Warehouse** ## **Overview** Ethiomed Data Warehouse is a scalable data infrastructure designed to collect, process, and analyze data on Ethiopian medical businesses. This project integrates **web and Telegram scraping**, **ETL/ELT pipelines**, and **YOLO-based object detection** to provide a robust platform for data-driven insights. ## **Features** ✅ **Automated Web & Telegram Scraping** – Collects real-time medical business data ✅ **ETL/ELT Pipelines** – Cleans, transforms, and loads data into the warehouse ✅ **YOLO-Based Object Detection** – Extracts visual insights from collected images ✅ **Scalable Data Warehouse** – Efficient storage and querying of structured data ✅ **Data Integration & Enrichment** – Enhances datasets for better analytics ## **Tech Stack** - **Programming Languages:** Python, SQL - **Scraping:** Scrapy, Selenium, Telethon (for Telegram) - **ETL/ELT:** Apache Airflow, Pandas, dbt (Data Build Tool) - **Data Warehouse:** PostgreSQL, BigQuery, Snowflake (configurable) - **Object Detection:** YOLOv8 (Ultralytics) - **Cloud & Storage:** AWS S3, Google Cloud Storage ## **Architecture** 1. **Data Collection:** Scrapes data from websites and Telegram channels 2. **Data Processing:** Cleans, normalizes, and structures data using ETL/ELT 3. **Object Detection:** Uses YOLO to detect medical-related objects in images 4. **Data Warehousing:** Stores processed data for fast querying and analysis 5. **Data Analysis & Reporting:** Generates insights for decision-making ## **Setup & Installation** ### **Prerequisites** Ensure you have the following installed: - Python 3.8+ - PostgreSQL / BigQuery / Snowflake - Docker (optional but recommended for deployment) ### **Installation Steps** 1. Clone the repository: ```bash git clone github.com cd ethiomed-datahub ``` 2. Install dependencies: ```bash pip install -r requirements.txt ``` 3. Set up environment variables (`.env` file): ```plaintext DAT …