Aakashanil67/regrag-sa-finance
Domaine:
digital infrastructureType de record:
software
Créateur:
Aak
Hôte:
RAG assistant over South African financial regulation, with enforced citations, refusal, and a full eval harness
# regrag-sa-finance
A retrieval-augmented assistant that answers questions about South African financial regulation
(SARB prudential directives, the National Credit Act, FSCA conduct standards, IFRS 9) from a fixed
local corpus of 19 PDFs. Every factual claim in an answer carries an inline `[doc_id, p.X]`
citation checked against the pages the system actually retrieved, and the system refuses rather
than answers when the retrieved context cannot support a response. The eval harness is the point
of the project rather than a checkbox after the fact: chunking and retrieval decisions were made
by sweeping alternatives against a retrieval benchmark and a RAGAS-judged golden set, so the
numbers below are what that sweep produced, not what was assumed going in.
## Eval results
The change under test is `chunk_size=800` with cross-encoder reranking, replacing 500-token chunks
with no reranking. Both configurations were measured on the same 20-question retrieval benchmark
and the same golden set.
| | chunk size | rerank | hit-rate@5 | MRR |
|---|---|---|---|---|
| before | 500 | off | 85% (17/20) | 0.654 |
| **after** | **800** | **on** | **95% (19/20)** | **0.808** |
The hit-rate line is two questions on a 20-question benchmark, and the Wilson intervals (64 to 95
per cent, 76 to 99 per cent) overlap too heavily for that difference to count as established. MRR
is the better-powered signal in the same data, since it moves on where the right chunk ranks
rather than only on whether it cleared a cutoff.
RAGAS was run on the golden set's answerable items with `claude-haiku-4-5` as both generator and
judge. The two runs scored different item sets, since reranking pulled six previously refused
questions into real answers and pushed one the other way, so the raw means below are not directly
comparable on their own. The paired comparison on the 33 items common to both runs sits alongside
them for that reason.
| metric | before (all, n=34) | after (all, n=39) | paired Δ (n=33) | …
Similaires
Re-examining the Constitutional Court's Approach to the Property Question Since First National Bank of SA Ltd T/A Wesbank v Commissioner, South African Revenue Service; First National Bank of SA Ltd T/A Wesbank v Minister of Finance 2002 4 SA 768 (CC)Literasiya sa Artificial Intelligence at Kasanayan ng mga Mag-Aaral sa Akademikong Pagsulat sa Senyor HayskulAISTORYA: Mga Salaysay sa Paggamit ng Artificial Intelligence ng mga Guro sa Filipino sa Pagbuo ng Kagamitang PampagtuturoXolani90/bizbot-samavhungu2/cartorque-saAsavelaTshonisile/CIVICLINK-SA