YonatanMoges/data-warehouse

Project focuses on building a robust data scraping from Telegram channels related to Ethiopian medical businesses and transformation pipeline that implements monitoring and logging. # data-warehouse ## Data Scraping and Transformation Pipeline ## Table of Contents 1. Introduction 2. Project Structure 3. Setup and Installation 4. Usage 5. Object detection using YOLO 6. Data Exposure Using FastAPI ## Introduction This project focuses on building a robust data scraping and transformation pipeline. The pipeline extracts data from public Telegram channels related to Ethiopian medical businesses, cleans and transforms this data, and implements monitoring and logging systems to ensure data quality. ## Project Structure The repository is structured as follows: ``` bash ├── app # FastAPI application directory │ ├── main.py # Main application file │ ├── database.py # Database connection and configuration │ ├── models.py # Data models for SQLAlchemy │ ├── schemas.py # Pydantic schemas for data validation │ └── crud.py # CRUD operations ├── data │ ├── raw_data.csv # Raw data scraped from Telegram │ ├── images/ # Folder storing images scraped │ ├── last_processed_ids.json # Tracks the last processed message IDs for each channel │ └── scraping.log # Log file for tracking the scraping process ├── logs # Log directory for pipeline logs ├── my_project # DBT project directory ├── notebook # Jupyter notebooks │ ├── __init__.py │ ├── data_cleaning.ipynb # Notebook for data cleaning │ └── yolo_detection.ipynb # Notebook for YOLO object detection ├── scripts # Directory for Python scripts │ ├── __init__.py │ ├── data_cleaner.py # Script for cleaning and transforming scraped data │ └── yolo_detection.py # Script for object detection using YOLO ├── yolov5 # YOLOv5 cloned repository for object detection ├── .env # Environment variables ├── .gitignore …