kibmuikia/ke-katiba-digest

A Python processing pipeline for extracting, structuring, and serialising legal text from the Constitution of Kenya (2010) and associated statutory booklet PDFs into structured JSON and SQLite formats. ke-katiba-digest A modern, high-performance extraction and structuring pipeline for the Constitution of Kenya (2010). Architecture • Getting Started • Usage • Standards • Schema --- ## Overview `ke-katiba-digest` converts raw, multi-column statutory PDF documents into clean, structured AST representations. It parses legal hierarchies (**Chapter → Part → Article → Clause**) and serializes them into production-ready **JSON** and **SQLite** databases for downstream legal tech applications, search engines, and LLM indexing. --- ## Architecture & Data Flow ```mermaid flowchart TD A[PDF Sources: Constitution / Laws of Kenya] --> B[PDF Extraction Layer] subgraph Step 1: Raw Extraction & Layout B -->|extractText.py| C1[Raw Text .txt] B -->|extractTextWithLayout.py| C2[Layout JSON .json] B -->|extractTables.py| C3[Extracted Tables] B -->|convertToMarkdown.py| C4[Cleaned Markdown .md] end C4 --> D[build_katiba_digest.py] subgraph Step 2: Parsing & Structuring D -->|Regex Parser| E[Hierarchical AST: Chapter -> Part -> Article -> Clause] D -->|test_parser.py| F[Smoke Test & Regex Validation] end E -->|generate_output.py| G1[(SQLite Database: constitution_kenya_2010.db)] E -->|generate_output.py| G2[Structured JSON: constitution_kenya_2010.json] ``` --- ## Directory Structure ``` ke-katiba-digest/ ├── build_katiba_digest.py # Core parser: AST builder & text cleanup pipeline ├── convertToMarkdown.py # PDF -> Markdown conversion CLI ├── extractTables.py # PDF tabular data extraction module ├── extractText.py # Baseline text extraction CLI ├── extractTextWithLayout.py # Layout-aware PDF text and bounding box extractor ├── generate_output.py # Exporter to SQLite (.db) and structured JSON (.json) ├── pdf_utils.py # Shared PDF processing & PyMuPDF/pdfplumber utilities ├── renderPageToImage.py # Page rendering engine for inspection/visual audit ├── test_parser.py # CLI test suite f …