Azazh/Medical-Data-Warehouse

**About** Kara Solutions is building a data warehouse for Ethiopian medical business insights by scraping Telegram channels. Using Python (Telethon) and DBT, we extract, clean, transform, and store data efficiently. AI-powered analysis and object detection enable better decision-making in the medical sector. 🚀 # Ethiopian Medical Businesses Data Pipeline This project focuses on building a robust data pipeline for Ethiopian medical businesses by scraping data from Telegram channels, cleaning and transforming the data, and storing it in a data warehouse for analysis. The pipeline consists of two main tasks: 1. **Task 1: Data Scraping and Collection Pipeline** - Scrapes data from Telegram channels. 2. **Task 2: Data Cleaning and Transformation** - Cleans and transforms the scraped data using Python and DBT. ## **Table of Contents** 1. Project Overview 2. Repository Structure 3. Task 1: Data Scraping and Collection Pipeline 4. Task 2: Data Cleaning and Transformation 5. Setup and Installation 6. Usage 7. Challenges and Solutions 9. Contributing 10. License ## **Project Overview** The goal of this project is to build a data pipeline that: - Scrapes data from Telegram channels related to Ethiopian medical businesses. - Cleans and transforms the scraped data. - Stores the data in a PostgreSQL database for analysis. The pipeline is designed to be modular, scalable, and easy to maintain. ## **Task 1: Data Scraping and Collection Pipeline** ### **Objective** Scrape data from Telegram channels, including text and media, and store it in a structured format. ### **Implementation** - **Tools**: Python (`telethon`, `pandas`, `logging`), Telegram API. - **Steps**: 1. Set up Telegram API access using `API_ID` and `API_HASH`. 2. Scrape data from specified Telegram channels (e.g., DoctorsET, Chemed). 3. Store raw data in JSON files and media files in a structured directory. 4. Log all activities for monitoring and debugging. ### **Output** - Raw data stored in `raw_data/` directory. - Media files stored in `raw_data/media/`. - Logs stored in `scraping.log`. ## **Task 2: Data Cleaning and Transformation** ### **Objective** Clean and transform the scraped data to ensure consistency, remove duplicates, and prepare it for analysis. ### **Implementation** - **Tools**: Python (`pa …