This note covers Automated Regulatory Compliance Question Answering in Financial Services with Domain-Adapted RAG, by Tobias Deusser with seven co-authors (arXiv:2609.30009, posted 24 September 2026, https://arxiv.org/abs/2609.30009). It is a preprint, currently under review, and it is not peer reviewed.

What the paper says

The task is to answer compliance questions over dense, often amended rulebooks. The authors say confidential documents often force firms to run compact models on their own hardware. They use models of 2B to 12B parameters at 4-bit quantization. Data is ObliQA, built from Abu Dhabi Global Market rulebooks, with 13,705 passages from 40 documents. Open Australian LegalQA is a second corpus, used as an out-of-domain control.

The published ObliQA splits share source documents across train, validation, and test. The authors re-split at document level, which removes 989 question-passage pairs. Their numbers are not comparable to published ObliQA results. The retriever starts from LegalBERT. Recall@10 on ObliQA starts at 0.256 for LegalBERT alone. It moves to 0.495 after the entailment step, to 0.732 after the contrastive step, and to 0.774 after BM25 fusion.

Baselines on the same split are BM25 at 0.678 and E5-large-v2 at 0.758. The tuned hybrid retriever reaches 0.774.

For the generator, the authors compare direct prompts with RAFT-LoRA adaptation on five compact models. RAFT-LoRA raised the RePASs score for the three models they adapted. Qwen2.5-7B moved from 0.668 to 0.725. Teuken-7B moved from 0.673 to 0.713. Gemma-2-2B moved from 0.594 to 0.666.

Adaptation of two models was unstable, so they are excluded. The adapters trained on ObliQA mostly did not transfer to Australian case law, and only Teuken improved, from 0.559 to 0.586. In the closed-book control, models with no passages scored within 0.011 RePASs of the full pipeline. The authors say the closed-book answers read as plausible regulatory prose, misstate obligations, omit the conditions that bind, and cite nothing. The authors conclude that RePASs alone cannot validate a compliance system. They say the main benefit of retrieval in their work is attributability, so an answer traces to a specific provision.

What it does not show

All reported numbers come from the validation split. The two-document ObliQA test split was held out and not used. The tuned retriever leads on Recall@10, 0.774 against 0.758. E5-large-v2 is marginally higher on MRR@10, 0.595 against 0.594. The authors call the two equivalent on first-hit rank and different at greater depth.

Generation is scored on 150 questions per configuration. Differences of a few thousandths are within noise. The closed-book runs cover 132 and 89 questions. The adaptation targets are outputs of GPT-4o-mini, not human answers. The adaptation sets hold 150 to 200 instances, under 1 percent of the questions in the train split.

The generation gain is a gain in RePASs. The authors say the gain does not show that answers rest on the passages.

Both corpora are English and public, and internal policy documents, tables, and multilingual rulebooks are untested. No Indonesian rules, live compliance workflows, or human expert evaluations appear. The re-split drops multi-passage questions, so the data tilts toward single-passage questions.

Why it matters in payments

The authors say the main benefit of retrieval in their setting is attributability: an answer traces to a specific provision. I know the same demand from the audit side.

In my PJP readiness case study, I created an evidence mapping approach that links each critical step to expected evidence, then marks whether it exists and what remediation is needed. I also standardized what counts as acceptable evidence by scenario type. The submission is under review.

An answer that cites nothing cannot be checked. An audit file without a link to its evidence cannot be checked either. My next step in that case study is to make evidence readiness a product requirement, so the link exists by design.