tedoaba/KAIM-W7

Building a Data Warehouse to Store Data on Ethiopian Medical Business Data Scraped from Telegram Channels # Building a Data Warehouse to Store Data on Ethiopian Medical Business Data Scraped from Telegram Channels ## KAIM Week 7 Challenges ## Project Overview This project aims to create a **data warehouse** for Ethiopian medical businesses by scraping relevant data from public **Telegram channels** and analyzing images through **object detection** using the **YOLO (You Only Look Once)** algorithm. The system includes processes for **data scraping**, **data cleaning**, **data transformation**, and **data storage**, as well as providing **API access** to the processed data. ### Key Objectives: 1. **Scraping Images from Telegram Channels**: Scrape images and metadata from specified channels using the Telegram API. 2. **Data Warehousing**: Store scraped images and their metadata in a relational database. 3. **Object Detection Preparation**: Set up data for object detection, ensuring proper storage and accessibility. 4. **Data Transformation**: Use DBT (Data Build Tool) to transform the stored data for object detection and further processing. 5. **API Development**: Develop an API to expose processed data for real-time insights and analysis. ## Table of Contents - Project Overview - Requirements - Setup Instructions - Task Breakdown - Task 1: Telegram Scraping - Task 2: Data Warehousing - Task 3: Object Detection - Task 4: Data Transformation with DBT - Task 5: API Development - Project Structure - Challenges and Solutions ## Requirements - **Python 3.x** - **Telethon** for Telegram API access - **SQLAlchemy** for database management - **PostgreSQL** or **SQLite** for data warehousing - **Pillow (PIL)** for image processing - **DBT (Data Build Tool)** for data transformation - **YOLOv5** (for object detection in future tasks) ## Task Breakdown ### Task 1: Telegram Scraping #### Overview: This task focuses on scraping images from Telegram channels using the **Telethon** library. Images are downloaded into a local folder, and metadata is collected for each image, …