Khayo-7/Kira-Solutions-Ethiopian-Medical-Businesses-Data-Warehouse-W7

# 10Academy-Kifiya-Week-7 # Building a Scalable Data Pipeline for Ethiopian Medical Businesses: ## Introduction In the rapidly evolving landscape of Ethiopian e-commerce, structured and accessible data is a key enabler for informed decision-making. At **Kara Solutions**, embarking on a mission to build a **comprehensive data warehouse** for Ethiopian medical businesses by leveraging data scraped from **Telegram channels**, the approach focuses on creating a **scalable, modular, and efficient pipeline**, ensuring seamless data collection, transformation, and analysis. --- ## Task 1: Data Scraping from Telegram ### **Objective:** The goal of this task was to **collect raw data from Telegram channels** where medical businesses actively post their products, services, and contact details. Given the decentralized nature of business listings in Ethiopia, Telegram serves as a crucial data source. ### **Approach:** A **scalable Telegram scraper** was developed using **Telethon**, an asynchronous Python library for interacting with the Telegram API. The scraper extracts messages from targeted channels, including: - **Business Listings** (clinic and pharmacy details, supplier information) - **Product Advertisements** (medications, equipment, medical supplies) - **Contact Information** (phone numbers, addresses, website links) - **Media Attachments** (images, PDFs, and videos related to the listings) ### **Key Features Implemented:** ✅ **Asynchronous scraping** for high efficiency and reduced API rate limitations ✅ **Storage in MongoDB**, ensuring structured collection and retrieval ✅ **Metadata extraction**, including timestamps, message IDs, and sender details ✅ **Handling media files**, downloading and storing paths for images and videos ✅ **Filtering relevant messages** based on keywords and patterns ### **Challenges Faced & Solutions:** **1. API Rate Limits:** implemented adaptive request throttling to avoid Telegram restrictions. **2. Unstructured Data:** Message …