maureen-githaiga/kenya-news-data-pipeline

End-to-end data pipeline for ingesting, processing, and analyzing Kenyan news articles using modern data engineering tools. # 🇰🇪 Kenya News Articles Analytics Pipeline ## Problem Statement Kenya generates thousands of daily news articles capturing critical political, social, and economic events. However, this unstructured text makes it difficult to extract insights like key topics, sentiment trends, and named entities. ## Solution This project solves this problem by establishing an automated ELT pipeline that: - **Extracts** raw, news data. - **Enriches** with NLP features (sentiment, entities, language detection, topic modeling). - **Transforms** into a **Star Schema designed and materialized through dbt, inside BigQuery**. - Visualizes insights in Metabase. ## Dataset The project uses a Kaggle dataset of over 5,700 web-scraped Kenyan news articles published between 2018 and 2025. Each record contains the article’s title, publication date, author, source and content. ## Tools - **Python** – Used for data ingestion, processing, and NLP enrichment. - **Terraform** – Defines and provisions all GCP and cloud infrastructure resources. - **Google Cloud Platform (GCP)** – Provides cloud services: BigQuery (Data Warehouse), Cloud Storage (GCS) (Data Lake), and IAM. - **dbt (Data Build Tool)** – For transforming and modelling data into a Star Schema. - **Kestra** – Workflow orchestration. - **Docker** – Containerization platform for orchestrator (Kestra) and visualisation (Metabase). - **Metabase** – Dashboard. ## Architecture .png) ## Star Schema Design The warehouse follows a **Star Schema** design for optimized analytics: - **Fact Table**: - `fct_articles`: contains metrics like article_id, published_date, sentiment, polarity, subjectivity, topic_id, author_id, source_id. - **Dimension Tables**: - `dim_authors`: maps author_id → author_name. - `dim_sources`: maps source_id → source_name. - `dim_topics`: maps topic_id → topic_label and keywords. This design allows efficient joins and flexible BI queries. ## Dashboard A simple dashboard built to visualize key insights from the Kenya news …