Hunegn/telegram-medical-business-scraper

This repository contains scripts for scraping public Telegram channels related to Ethiopian medical businesses. It extracts both text messages and images for further analysis, including Natural Language Processing (NLP) and Object Detection. # Ethiopian Medical Business Data Scraper ## Project Overview This project focuses on scraping and analyzing data from Ethiopian medical business-related Telegram channels. The extracted data will be used to build a data warehouse, conduct object detection on images, and expose the data through a FastAPI service. ## Features - **Telegram Scraping:** Extract data from various Ethiopian medical business Telegram channels. - **Image Scraping & Processing:** Collect and preprocess images for object detection. - **Data Cleaning & Transformation:** Process extracted data into structured formats. - **Data Warehouse:** Store and manage the cleaned data efficiently. - **API Integration:** Serve the data through a FastAPI-based REST API. ## Data Sources We will scrape data from the following Telegram channels: - DoctorsET - Chemed Telegram Channel - Yetenaweg - EAHCI - Additional sources from et.tgstat.com ## Folder Structure ```plaintext Ethiopian-Medical-Data-Scraper/ │-- data/ # Raw and processed data storage │ ├── raw/ # Unprocessed scraped data │ ├── processed/ # Cleaned and transformed data │ ├── images/ # Downloaded images │ ├── database/ # Data warehouse (SQLite/PostgreSQL) │ │-- scripts/ # Python scripts for various tasks │ ├── telegram_scraper.py # Extracts data from Telegram channels │ ├── image_scraper.py # Scrapes images from Telegram │ ├── data_cleaning.py # Cleans and structures the scraped data │ ├── object_detection.py # Applies object detection models on images │ ├── data_warehouse.py # Manages the structured data warehouse │ ├── api_service.py # FastAPI service to expose the data │ │-- models/ # Object detection models │-- notebooks/ # Jupyter notebooks for data exploration & analysis │-- logs/ # Logging for t …