declerke/BizPulse-Kenya

Kenya business news sentiment pipeline: Snowflake + MLflow + FinBERT + Evidence.dev + Airflow 3 # 📰 BizPulse Kenya: Business News Sentiment & Economic Intelligence **BizPulse Kenya** is a production-grade data engineering and AI solution designed to bridge the gap between raw Kenya business news and structured economic intelligence. It ingests daily articles from Business Daily Africa and Standard Media, scores sentiment using three competing NLP models tracked in MLflow, stores all data in a Snowflake cloud warehouse, and generates weekly AI briefings via a free LLM — all orchestrated by Apache Airflow 3. --- ## 🎯 Project Goal Kenya's business press publishes hundreds of articles daily that collectively signal economic direction — but no structured data layer exists to quantify that signal. BizPulse Kenya solves this by running three sentiment models (FinBERT, VADER, TextBlob) against every article, logging each model's performance in MLflow, registering the champion (FinBERT) in a model registry, and synthesising the week's economic narrative into a professional briefing using LLaMA 3.1 8B via the free Groq API. The result is a live Evidence.dev dashboard tracking sentiment, forex trends, and CBK interest rates — with weekly AI-written reports anyone can read. --- ## 🧬 System Architecture 1. **Ingestion Layer** — `feedparser` parses RSS feeds from Business Daily Africa and Standard Media; `requests` + `BeautifulSoup4` scrape CBK indicative forex rates and interest rates daily 2. **Cloud Warehouse** — Snowflake (AWS af-south-1, Cape Town) stores raw articles and CBK data in the `RAW` schema; dbt transforms produce `STAGING` views and `MART` tables 3. **MLflow Experiment Tracking** — Three sentiment models run in parallel per DAG trigger; metrics (positive/negative/neutral counts, avg confidence, processing time) are logged per model; FinBERT is registered as the champion in the Model Registry 4. **Transformation Layer** — dbt-snowflake materialises 4 models (3 staging views + 1 mart table); 14 data quality tests enforce freshness and schema contracts; …