Starias22/Africa-News-API

# Africa-News-API ## Table of Contents 1. Summary 2. Features 3. Tools 4. Architecture 5. Database Design 6. Installation - Prerequisites - Setup Steps 7. Usage 8. API Endpoints 9. ETL Pipeline 10. Future Work 11. Contributing 12. License 13. Contact Information ## Summary The Africa-News-API is a personal project designed to aggregate news from various African media platforms and existing news APIs into a centralized database. News articles are retrieved, transformed, and stored hourly, making them accessible to users through dedicated API endpoints, facilitating easy access to news events across the continent. ## Features - Aggregates news from multiple African media websites and news APIs. - Utilizes fake user agents to avoid blocking and enhance the success rate of web scraping operations. - Efficiently processes large volumes of news articles using Apache Spark. - Stores news articles in a PostgreSQL database. - Provides a FastAPI-based API for accessing news data. - Automated ETL (Extract, Transform, Load) pipeline using Apache Airflow. - Supports containerized deployment with Docker and Docker Compose. ## Tools - **PgModeler**: For database modeling and design. - **PostgreSQL**: Relational database management system for storing news articles. - **SQLAlchemy**: For database interaction and management, serving as an Object-Relational Mapping (ORM) tool. - **Python**: Primary programming language for data retrieval and processing. - **Selenium**: For web scraping news articles from media websites. - **Apache Spark**: For processing and analyzing large volumes of news data efficiently. - **Apache Airflow**: To orchestrate and automate data workflows and ETL processes. - **FastAPI**: To create a performant and easy-to-use API for accessing news data. - **Docker**: For containerizing the application to ensure consistency across different environments. - **Docker Compose**: For defining and managing multi-container Docker applications, simplifying the setup and …