Filimon-H/ethio-med-telegram-analytics

End-to-end data pipeline analyzing Ethiopian medical Telegram channels. Includes data scraping, warehouse modeling with dbt, image detection (YOLOv8), and FastAPI analytics. Built with Docker, PostgreSQL, and Dagster orchestration." # 🏥 ETHIO-MED-TELEGRAM-ANALYTICS **Containerized analytics platform** for Ethiopian medical Telegram data 📦🇪🇹 End-to-end pipeline for NLP/CV tasks: Named Entity Recognition (NER) & YOLOv8 image detection --- ## 🧠 Project Overview Robust **ELT pipeline** ingesting Amharic Telegram messages from Ethiopian medical channels. Processes data for: - 🧠 Amharic NER (Named Entity Recognition) - 👁️ YOLOv8 medical object detection - 📊 Business analytics API --- ## 🔧 Tech Stack | Layer | Tool | |-------------------|-------------------------------| | 🐘 **Database** | PostgreSQL 14 + pgAdmin | | 🧱 **Transform** | dbt (Data Build Tool) | | 📤 **Ingestion** | Telethon + Custom Python | | 🔍 **API** | FastAPI + Pydantic | | 🤖 **ML Models** | YOLOv8, HuggingFace Transformers | | ⚙️ **Orchestration** | Dagster | | 🐳 **Infra** | Docker + Docker Compose | --- ## 📂 Directory Structure ├── data/ # Telegram data │ ├── raw/ # Raw JSONs │ └── preprocessed/ # Cleaned data ├── ethio_api/ # FastAPI app ├── ethio_pipeline/ # Dagster jobs ├── telegram_dbt_project/ # dbt models ├── scripts/ # Data scripts ├── notebooks/ # EDA/ML experiments ├── .env # Environment secrets ├── docker-compose.yml # Multi-container setup ├── requirements.txt # Dependencies └── README.md ## 🚀 Quickstart Guide ### 1. Clone & Configure ```bash git clone github.com cd ETHIO-MED-TELEGRAM-ANALYTICS cp .env.example .env 📝 Edit .env with PostgreSQL and Telegram credentials 2. Launch Containers docker compose up --build 3. Scrape Messages python scripts/scrape_telegram.py --channel @cheMed123 --limit 1000 4. Load Raw Data python scripts/load_raw_json.py --input data/raw/telegram_messages/YYYY-MM-DD/ 5. Run dBT Transformations docker exec -w /app/telegram_dbt_project -it telegram_app dbt build 6. Serve Documentation (Optional) docker exec -w /app/te …