kaleabb266/ethiopian-medical-data-warehouse

# **Ethiopian Medical Data Warehouse** ## **Project Overview** This project is a **data warehouse** built to store, clean, transform, and analyze **Ethiopian medical business data** scraped from Telegram channels. The data pipeline includes: ✅ **Web Scraping** (Telegram API & Telethon)\ ✅ **Data Cleaning & Transformation** (Pandas & DBT)\ ✅ **Object Detection** (YOLOv5 for image processing)\ ✅ **Database Storage** (PostgreSQL for structured data)\ ✅ **API Development** (FastAPI for data access) --- ## **Tech Stack** - **Programming Language**: Python 🐍 - **Data Scraping**: Telethon, BeautifulSoup, Scrapy - **Data Processing**: Pandas, NumPy, DBT (Data Build Tool) - **Database**: PostgreSQL - **Object Detection**: YOLOv5, OpenCV, Torch - **API Development**: FastAPI, Uvicorn --- ## **Project Structure** ``` ethiopian-medical-data-warehouse/ │── data_scraping/ │ ├── scraper.py # Telegram data scraper │── data_cleaning/ │ ├── data_cleaning.py # Cleans and transforms scraped data │── dbt_models/ │ ├── transformations/ │ │ ├── cleaned_messages.sql # DBT model for data transformation │── object_detection/ │ ├── detector.py # YOLOv5 object detection script │── api/ │ ├── main.py # FastAPI application │── database/ │ ├── load_to_database.py # Inserts cleaned data into PostgreSQL │── notebooks/ │ ├── data_cleaning.ipynb # Jupyter Notebook for exploratory data analysis │── requirements.txt # Project dependencies │── README.md # Project documentation ``` --- ## **Installation & Setup** ### **1️⃣ Clone the Repository** ```bash git clone github.com cd ethiopian-medical-data-warehouse ``` ### **2️⃣ Set Up a Virtual Environment** ```bash python -m venv env source env/bin/activate # macOS/Linux env\Scripts\activate # Windows ``` ### **3️⃣ Install Dependencies** ```bash pip install -r requirements.txt ``` ### **4️⃣ Set Up Datab …