habeshaethiopia/ethiomed-datahub
Domaine:
healthcareType de record:
software
Créateur:
hab
Hôte:
A scalable data warehouse for Ethiopian medical businesses, integrating web and Telegram scraping, ETL/ELT pipelines, and YOLO-based object detection. This project enables efficient data collection, transformation, and analysis to uncover insights and trends in the medical sector.
# **Ethiomed Data Warehouse**
## **Overview**
Ethiomed Data Warehouse is a scalable data infrastructure designed to collect, process, and analyze data on Ethiopian medical businesses. This project integrates **web and Telegram scraping**, **ETL/ELT pipelines**, and **YOLO-based object detection** to provide a robust platform for data-driven insights.
## **Features**
✅ **Automated Web & Telegram Scraping** – Collects real-time medical business data
✅ **ETL/ELT Pipelines** – Cleans, transforms, and loads data into the warehouse
✅ **YOLO-Based Object Detection** – Extracts visual insights from collected images
✅ **Scalable Data Warehouse** – Efficient storage and querying of structured data
✅ **Data Integration & Enrichment** – Enhances datasets for better analytics
## **Tech Stack**
- **Programming Languages:** Python, SQL
- **Scraping:** Scrapy, Selenium, Telethon (for Telegram)
- **ETL/ELT:** Apache Airflow, Pandas, dbt (Data Build Tool)
- **Data Warehouse:** PostgreSQL, BigQuery, Snowflake (configurable)
- **Object Detection:** YOLOv8 (Ultralytics)
- **Cloud & Storage:** AWS S3, Google Cloud Storage
## **Architecture**
1. **Data Collection:** Scrapes data from websites and Telegram channels
2. **Data Processing:** Cleans, normalizes, and structures data using ETL/ELT
3. **Object Detection:** Uses YOLO to detect medical-related objects in images
4. **Data Warehousing:** Stores processed data for fast querying and analysis
5. **Data Analysis & Reporting:** Generates insights for decision-making
## **Setup & Installation**
### **Prerequisites**
Ensure you have the following installed:
- Python 3.8+
- PostgreSQL / BigQuery / Snowflake
- Docker (optional but recommended for deployment)
### **Installation Steps**
1. Clone the repository:
```bash
git clone github.com
cd ethiomed-datahub
```
2. Install dependencies:
```bash
pip install -r requirements.txt
```
3. Set up environment variables (`.env` file):
```plaintext
DAT …