declerke/Kenya-Doc-Intelligence

Automated NLP pipeline for CBK and Kenya National Treasury PDF documents - Airflow 3.0, pdfplumber, spaCy, HuggingFace, dbt, PostgreSQL, Streamlit # 📄 Kenya Financial Document Intelligence: Automated NLP Pipeline for CBK & Treasury Reports **A production-grade document intelligence platform that automatically discovers, downloads, and analyses official Kenyan financial policy PDFs — bridging the gap between raw government archives and structured, queryable insights on monetary policy, fiscal strategy, and economic trends. An Airflow 3.0 DAG orchestrates the full lifecycle across 87 documents, 2,490 extracted pages, and 40,801 entity mentions.** --- ## 🎯 Project Goal Kenya's Central Bank (CBK) and National Treasury publish dozens of financial policy documents annually — MPC press releases, annual reports, budget statements, and financial stability reports — but these remain locked in unstructured PDFs, inaccessible to systematic analysis without significant manual effort. This pipeline automates the full document intelligence lifecycle: from web discovery and PDF ingestion through a four-model NLP chain to a Streamlit dashboard that exposes searchable, time-series insights across the full corpus of Kenyan financial policy documents. ## 🧬 System Architecture 1. **Discovery Layer** — **BeautifulSoup4** scrapes CBK and Treasury listing pages supplemented by a seeded baseline of 12 verified PDF URLs, deduplicating by URL before insertion 2. **Ingestion Layer** — **requests** downloads PDFs to a named Docker volume, tracking download state per document and skipping already-processed files on reruns 3. **Extraction Layer** — **pdfplumber** extracts page-level text, storing each page separately with word counts for downstream NLP chunking 4. **NLP Enrichment Layer** — **HuggingFace Transformers** (abstractive summarisation + sentiment classification), **spaCy en_core_web_sm** (NER across 5 entity types), **scikit-learn TF-IDF** (corpus-level keyword extraction); all models run CPU-only inside Docker 5. **Transformation Layer** — **dbt-postgres** 7-model pipeline (staging → intermediate → 4 mart tables) with 55 d …