AISWARYA-NANDAKUMAR/epistemic-factkg-benchmark
Domain:
natural language processingRecord type:
paper
Creator:
AIS
Host:
When Shortcut-Breaking Splits Leak: a source-trust-isolating benchmark for fact verification (Togo AI Labs)
# Beyond Binary Trust: A Source-Trust-Isolating Benchmark for Continuous Epistemic Reasoning in Fact Verification
**Dheeraj Karki** — Independent Researcher (dheerajkarki1790@gmail.com)
**Aiswarya Konavoor** — Togo AI Labs (aiswarya@togolabs.ai)
A diagnosis-and-benchmark paper on measuring **source-trust reasoning** in fact verification.
📄 **Read the paper (PDF)** · 🌐 **Project website** · ⚙️ **System repo** · 🧪 **Benchmark repo**
---
## The one-paragraph version
To claim a fact-verification model uses *source trust*, you need a test only a trust-using model can pass.
The natural construction, pairing the same claim with a high-trust and a low-trust source, **secretly
leaks the verdict**: the low-trust evidence is written in a different register, so a model reads the style
instead of the source. We quantify the leak, show that AVeriTeC *accuracy* of 0.665 is just the
majority-class prior (macro-F1 0.265), build a **trust-isolating benchmark** that admits no detectable
non-trust path to the label, and report an **inversion**: a trust-blind model that *wins* on the leaky
split (0.996 accuracy) collapses to chance on the corrected one (macro-F1 0.385 ± 0.007), while a
trust-aware model wins (v2-HGNN macro-F1 0.918 ± 0.017). Mid-tier boundary probes further show that
access to the trust scalar alone is insufficient — trust must be reasoned over continuously.
## The inversion (headline result)
| Setting | Metric | Trust-blind | Trust-aware (v2-HGNN) | v3-NLI |
|---|---|---|---|---|
| Original split (leaky) | accuracy | **0.996** | 0.889 | — |
| Corrected, LogReg probe | macro-F1 | 0.389 | **0.882** | — |
| Corrected, GNN (3 runs) | macro-F1 | 0.385 ± 0.007 | **0.918 ± 0.017** | 0.907 ± 0.019 |
| Always-NEE floor | macro-F1 | 0.250 | — | — |
| Mid-tier probes (LogReg) | macro-F1 | 0.016 | — | 0.204 |
Source trust helps **only once the split stops leaking**.
## Repository layout
```
Justfile # just probe / e …