ElbetelTaye/Building-a-data-warehouse

This project aims to build a data warehouse to store and analyze data from Ethiopian medical businesses, collected from web scraping and Telegram channels. By integrating object detection with YOLO, the warehouse will enhance data analysis and provide valuable insights for informed business decisions. # Data Cleaning and Transformation for Ethiopian Medical Businesses ## Project Overview This project focuses on creating a data warehouse to store and analyze information on Ethiopian medical businesses. The data is sourced through web scraping and Telegram channels, with integrated YOLO (You Only Look Once) object detection capabilities to enhance data analysis. The warehouse enables better insights into business trends, patterns, and correlations, facilitating informed decision-making. ## Notebooks ### data_preprocessing.ipynb This notebook handles the following tasks: - **Data Loading**: Loads JSON files containing Ethiopian medical business data from a specified directory. - **Data Cleaning**: - Removes duplicate entries. - Fills in missing values where possible and drops rows missing essential data. - Standardizes formats, ensuring consistency, especially in date and text fields. - Validates cleaned data to meet quality standards. - **Database Integration**: Stores cleaned data in a PostgreSQL database using SQLAlchemy. - **Image Resizing for YOLO**: Resizes images for YOLO object detection. - **Error Handling and Logging**: Tracks errors and logs progress in `data_cleaning_task2.log`. ### object_detection_yolo.ipynb - **Data Validation**: Checks the paths for training images, labels, and `data.yaml` configuration to ensure they are set correctly. - **Training Setup**: Configures YOLOv5 training parameters, such as `epochs`, `batch_size`, and `img_size`. - **YOLOv5 Training Execution**: Runs YOLOv5 training using a subprocess with the necessary dataset paths and configuration. - **Model Weight Management**: Saves the best and final weights from training as `labeled_yolov5_model.pt`. - **Database Integration**: Saves object detection results in the warehouse database. ## Setup Instructions Install the dependencies using: ```bash pip install -r requirements.txt ``` ## DBT Integration DBT (Data Build Tool) is used to manage data transformations and ensur …