nahomWM/Ethiopian-Related-news-scraping-engnie

A powerful, multilingual news scraping engine designed to collect and process news articles from 80+ sources across Arabic, French, Swahili, and Somali languages. Built for RAG systems # News Scraping Engine A powerful, multilingual news scraping engine designed to collect and process news articles from 80+ sources across Arabic, French, Swahili, and Somali languages. Built for RAG (Retrieval-Augmented Generation) systems with support for both web scraping and Telegram channel monitoring. ## Author **Nahom_WM** ## Features - 🌍 **Multilingual Support**: Scrapes news in Arabic (ar), French (fr), Swahili (sw), and Somali (so) - 📰 **80+ News Sources**: Comprehensive coverage from major news outlets across Africa and the Middle East - 📡 **Telegram Integration**: Monitor and scrape content from Telegram channels - 🎯 **Smart Filtering**: Relevance filtering using multilingual anchor terms - 💾 **Dual Storage**: Local JSON files and MongoDB Atlas cloud storage - ⚡ **Async Processing**: High-performance concurrent scraping with semaphore-based rate limiting - 🔄 **Source Synchronization**: Smart sync between local and cloud storage - 📊 **RSS & HTML Parsing**: Supports both RSS feeds and direct HTML link extraction - 🕒 **Date Extraction**: Intelligent article date parsing from URLs and content ## Data Sources The engine aggregates news from diverse sources including: - **Somali**: BBC Somali, VOA Somali, Garowe Online, Goobjoog, Caasimada, and 15+ more - **Arabic**: Al Jazeera, BBC Arabic, Sky News Arabia, Al Arabiya, France24 Arabic, and 30+ more - **French**: RFI, France24, Jeune Afrique, Le Monde Afrique, TV5Monde, and 15+ more - **Swahili**: BBC Swahili, DW Swahili, VOA Swahili, Taifa Leo, and more Full source list available in `src/config.py` ## Architecture ``` news-scraping-engine/ ├── main.py # Main entry point and pipeline orchestration ├── requirements.txt # Python dependencies ├── .env # Environment configuration (not in repo) ├── src/ │ ├── config.py # News sources and anchor terms configuration │ ├── scraper.py # Core web scraping logic (RSS + HTML) …