mihretgold/Ethiopian-Medical-Business-Data-Ware-house
Domain:
healthcareCreator:
mih
Host:
# Ethiopian Medical Business Data Warehouse
## Overview
This project involves building a data warehouse to store and analyze data on Ethiopian medical businesses scraped from Telegram channels. The project includes developing a data scraping and collection pipeline, cleaning and transforming the data, integrating object detection using YOLO, and designing and implementing the data warehouse.
## Table of Contents
- Business Need
- Project Structure
- Technologies Used
- Setup and Installation
- Data Scraping and Collection
- Data Cleaning and Transformation
- Object Detection using YOLO
- API Development with FastAPI
- Learning Outcomes
- Competency Mapping
- References
## Business Need
Kara Solutions, a leading data science company, requires a robust and scalable data warehouse to store and analyze data on Ethiopian medical businesses. The centralized data storage allows comprehensive analysis to uncover valuable insights, enabling better decision-making and efficient querying and reporting.
## Project Structure
```plaintext
my_project/
├── main.py
├── database.py
├── models.py
├── schemas.py
└── crud.py
```
## Technologies Used
- Python
- BeautifulSoup
- Scrapy
- Selenium
- YOLO (You Only Look Once)
- PostgreSQL
- FastAPI
- DBT (Data Build Tool)
- SQLAlchemy
- Pydantic
## Setup and Installation
1. **Clone the repository:**
```sh
git clone github.com
cd your-repo-name
```
2. **Install dependencies:**
```sh
pip install -r requirements.txt
```
3. **Set up the database:**
Configure your database connection in `database.py`.
## Data Scraping and Collection
1. **Telegram Scraping:**
- Use the Telegram API or custom scripts to extract data from public Telegram channels.
- Example channels:
- DoctorsET
- Chemed Telegram Channel
- EAHCI
2. **Image and Data Scraping:**
- Collect images from channels like Chemed Telegram Channel for object detection.
- Use Python packages like `telethon` for extraction.
## Data Cleanin …