nebiyuephrata/rataz-tech-refinery-os

Open-source, production-grade document intelligence engine by Rataz Tech for deterministic extraction, normalization, provenance-preserving chunking, semantic indexing, and multilingual querying (including Amharic), with graceful degradation and configurable escalation paths on fully open infrastructure. # Rataz Tech Refinery-OS Open-source document intelligence engine for deterministic extraction, provenance-preserving transformation, and auditable semantic retrieval. ## Stack (Free & Minimal) | Layer | Tool | |---|---| | PDF reading | pdfplumber / PyMuPDF (implemented for Tier A) | | Layout extraction | Docling / MinerU adapters | | OCR | Tesseract adapter | | Table extraction | Camelot adapter | | Embeddings | BGE-small (planned) | | Vector store | FAISS local (planned) | | DB | SQLite (planned persistent audit) | | API | FastAPI | ## Submission Artifacts - DOMAIN_NOTES.md - Architecture Diagram - Code Skeletons + Prompt - Cost Estimation - Test Results ## MVP Features - Five executable stages: triage, structure extraction, semantic chunking, page index builder, query interface. - Strategy + Factory + Adapter patterns. - Config-driven extraction triage and fallback chain. - Pydantic typed outputs across pipeline and API. - Trace IDs + request audit trail. - Persistent storage backends for audit/extraction history (`memory` and `sqlite`). - Amharic + English localization support. - PageIndex tree build and query (`/pageindex/{document_id}`, `/pageindex/query`). - Vector ingestion stores full per-chunk metadata (`chunk_type`, `page_refs`, `content_hash`, `parent_section`). - FactTable extraction persists numerical key-value facts into SQLite with SQL retrieval via `/query/structured`. ## Storage Rubric Evidence - Vector metadata ingestion: - /home/rata/Documents/Ephrata/work/10Acadamy/training/rataz-Wordz/src/rataz_tech/indexing/strategies.py - /home/rata/Documents/Ephrata/work/10Acadamy/training/rataz-Wordz/tests/test_indexing_metadata_ingestion.py - FactTable extractor + SQLite schema + SQL query path: - /home/rata/Documents/Ephrata/work/10Acadamy/training/rataz-Wordz/src/rataz_tech/indexing/facts.py - /home/rata/Documents/Ephrata/work/10Acadamy/training/rataz-Wordz/src/rataz_tech/api/services.py - /home/rata/Documents/Ephrata/work/10Acadamy/training/rat …