Generative AI in Banking Without Failing Model Risk Review
Getting a Language Model Through the Committee That Validates Models
Generative AI pilots in banking rarely fail on capability. They fail at the point where the model risk function asks for the validation evidence it asks for on every other model: a documented development process, a demonstrable performance measure, a stable relationship between inputs and outputs, and evidence of effective challenge. A language model produces none of that in the expected form, so the pilot stalls in review and eventually quietly closes.
The way through is not to argue that generative systems are exempt. It is to build the system so validation is possible, then present evidence in the shape the committee needs. Deploying generative AI banking model risk functions will approve is a design problem before it is a governance problem.
Why does generative AI break model validation practice?
Because validation assumes determinism, stable inputs, and backtestable performance, and generative systems provide none of them.
| Validation expectation | Traditional model | Generative system |
|---|---|---|
| Same input gives same output | Yes | No, unless constrained |
| Development data documented | Yes | Pre-training data undisclosed for most vendor models |
| Performance measurable on one metric | Usually | Requires task-specific evaluation with judgment |
| Backtesting against history | Standard | Often no historical equivalent exists |
| Sensitivity analysis on inputs | Tractable | Prompt sensitivity is real and hard to bound |
| Full inspection of internals | Possible | Not possible for a vendor-hosted model |
That table is the whole difficulty, and it is why an approach of promising the model is accurate will not survive contact with a validator. What does survive is demonstrating that the system, including everything around the model, produces acceptable outcomes with known failure modes and controls that catch them.
What does current supervisory guidance say?
In the United States, SR 26-2 replaced the long-standing SR 11-7 in April 2026 and takes a risk-based approach.
The Federal Reserve issued SR 26-2, Revised Guidance on Model Risk Management, on 17 April 2026. It replaces SR 11-7 from 2011 and SR 21-8 from 2021, emphasises a risk-based approach tailored to a banking organisation's model risk profile, and applies primarily to organisations above thirty billion dollars in assets.
There is an important catch, and it cuts against the assumption most teams start from. The companion interagency issuance, OCC Bulletin 2026-13 of the same date, states that the revised guidance explicitly excludes AI and generative AI models on the basis that they are novel and rapidly evolving, and that non-compliance with the guidance will not result in supervisory criticism. So the current model risk guidance is not the framework your generative system will be judged against, and waiting for it to be is not a strategy. What actually happens is that your own model risk policy, written when SR 11-7 was current, still applies internally, while examiners assess generative systems through consumer protection, fair lending, conduct, operational resilience, and third-party risk lenses instead. Plan for that combination rather than for a single tidy standard.
Does your model risk policy still reference guidance that was superseded in 2026?
Is a generative system a model for inventory purposes?
Usually yes where its output informs a business decision, and the classification should be explicit.
Teams sometimes try to avoid the model inventory by calling a system a tool or an assistant. That is a poor strategy, because an unregistered system whose output shapes credit, pricing, or customer treatment is precisely the finding an examiner or internal audit looks for. Register it, classify it by use, and let the classification drive proportionate requirements.
| Use pattern | Typical classification | Requirements |
|---|---|---|
| Drafting text a human edits and owns | Tool with model risk attributes | Evaluation, logging, human accountability |
| Summarising documents for a human decision | Model, low to moderate tier | Grounding, citation, sampling review |
| Extracting data that feeds a decision system | Model, moderate to high tier | Accuracy measurement, reconciliation, monitoring |
| Producing a recommendation or score | Model, high tier | Full validation, effective challenge, override tracking |
| Deciding autonomously | High tier and usually inadvisable initially | Extensive controls, strong case needed |
How do you validate a non-deterministic system?
With task-specific evaluation on real ground truth, repeated across runs, plus human sampling and adversarial testing.
Build a held-out evaluation set from real cases with a correct answer established independently of the model, ideally by the experts whose work the system supports. Then measure repeatedly rather than once, since variance across runs is itself a finding a validator needs. Add human review sampling in production with a defined rate and a documented reviewer standard, adversarial testing for prompt injection and manipulation, and drift monitoring on both inputs and outputs. The result is not a single accuracy number, it is a profile of behaviour with known failure modes, which is a more honest artefact than the point estimate traditional validation expects.
How do you build an evaluation set with real ground truth?
By having practitioners label real historical cases before the system is built, and holding that set back.
This is the unglamorous work that determines whether the project passes review, and it is usually underfunded. Take a representative sample of real cases including the awkward ones, have qualified people establish the correct output, document the labelling standard and disagreements between labellers, and keep the set out of any prompt tuning or few-shot examples. Refresh it periodically so it reflects current conditions. Teams that skip this end up demonstrating quality with anecdotes, which no validator accepts.
How do you handle a vendor model you cannot inspect?
By validating the system rather than the weights, and documenting the vendor dependency as a risk.
You will not get pre-training data, architecture detail, or internals for a hosted frontier model. What you can do is evaluate behaviour on your own tasks and data, document the vendor's own evaluations and their limits, pin model versions so a silent provider update does not invalidate your validation, and require notification of model changes contractually. Then record the concentration and change risk explicitly, since a provider updating a model underneath you is both a model risk and a third-party risk. Version pinning is the single most important technical control here, because uncontrolled model updates make every prior validation historic.
How does effective challenge work here?
By challenging the system boundary, the grounding, the evaluation design, and the human controls, rather than the model internals.
Effective challenge is about independent, competent scrutiny with real authority to require change, and that translates well to generative systems even without model transparency. An independent reviewer should test whether the use case boundary is respected, whether grounding data is complete and current, whether the evaluation set is representative and genuinely held out, whether failure modes were sought rather than avoided, and whether the human controls work under realistic workload rather than in a demonstration. Give the reviewer budget to run their own tests, because a challenge function that only reads documentation is not challenge. The governance framing for this is covered in this guide to AI model governance and explainability.
Which architecture choices reduce model risk?
Grounding, citation, structured output, deterministic guardrails, and human accountability at the decision point.
| Choice | Risk it reduces |
|---|---|
| Retrieval grounding on controlled sources | Unsupported assertions, stale knowledge |
| Mandatory citation to source passages | Unverifiable output, reviewer burden |
| Structured output schemas | Free-text ambiguity, downstream parsing errors |
| Deterministic validation of generated values | Arithmetic and reference errors reaching decisions |
| Explicit refusal and escalation paths | Silent guessing on out-of-scope questions |
| Version pinning for model and prompts | Unmonitored behaviour change |
| Human accountability for the output | Automation bias and unclear ownership |
Why is retrieval grounding a control rather than a feature?
Because it converts an unverifiable assertion into a checkable claim.
If the system answers from cited internal documents, a reviewer can verify the answer against the source in seconds, and a wrong answer becomes traceable to either a retrieval failure or a generation failure. That is exactly the diagnosability validation requires, and it also constrains the model's scope to material the institution controls. Measure retrieval quality separately from generation quality, since most production failures in these systems are retrieval failures wearing a generation costume. The explainability dimension is developed further in this guide to model explainability challenges.
Can a reviewer verify each generated statement against a cited source in seconds?
What does the NIST framework add?
A common structure for trustworthiness and a companion profile specific to generative systems.
The NIST AI Risk Management Framework 1.0, released on 26 January 2023, is a voluntary, consensus-developed framework for incorporating trustworthiness into the design, development, use, and evaluation of AI systems, and NIST released a Generative Artificial Intelligence Profile, NIST-AI-600-1, on 26 July 2024 to help organisations identify risks unique to generative AI and choose actions to manage them. Neither replaces supervisory model risk requirements, and both are useful for structuring the parts model risk guidance was not written for, particularly content risks and the operational controls around a generative system. Note that the AI RMF is itself under revision, so reference the version you used and date your assessment.
Which use cases pass review most easily?
Those with a human decision maker, controlled source material, and visible recoverable errors.
| Use case | Review difficulty | Why |
|---|---|---|
| Drafting internal documents from cited sources | Low | Human owns the output, errors visible |
| Summarising policy or procedure documents | Low | Grounded, verifiable, non-decisioning |
| Search and question answering over internal material | Low to moderate | Citation makes checking cheap |
| Extracting structured data from documents | Moderate | Accuracy measurable, reconciliation possible |
| Drafting customer communications | Moderate to high | Conduct and fair treatment implications |
| Producing credit or risk assessments | High | Directly decisioning, requires full validation |
| Autonomous customer-facing decisions | Very high | Rarely justified as a first deployment |
Start at the top of that table. The organisational capability built delivering low-difficulty use cases, including evaluation sets, logging, monitoring, and reviewer workflow, is what makes a high-difficulty case approvable later. Institutions that begin with a decisioning use case spend a year in review and deliver nothing.
What does the approval package contain?
Boundary, classification, data lineage, evaluation, failure modes, controls, monitoring, and named owners.
Write it as a document a validator can assess without a demonstration. State precisely what the system does and does not do, its inventory classification and tier, the grounding sources with their lineage and refresh cadence, the evaluation methodology, set construction, and results including variance, the failure modes you found and how each is controlled, the human controls with expected reviewer workload, the production monitoring plan with thresholds and owners, and the accountable business and technical owners. Add the vendor dependency and version pinning approach. That package is roughly the same shape as an ordinary model submission, which is the point: familiar structure with generative-specific content is far easier to approve than a novel format.
Which metrics should be monitored in production?
Grounding and citation rates, human override and correction rates, retrieval quality, refusal rates, and drift.
Monitor the share of outputs fully grounded in cited sources, since a falling rate means the system is drifting toward unsupported generation. Track human override and correction rates, which are the best available proxy for quality and also the signal for automation bias if they fall implausibly low. Measure retrieval quality independently. Watch refusal and escalation rates, because a collapse in refusals usually means the system has started answering things it should decline. Add input and output drift monitoring and periodic re-evaluation against the held-out set, run on a schedule rather than on suspicion. Those pipelines are ordinary machine learning operations work, as set out in this guide to MLOps for regulated workloads, and the responsible-deployment patterns are covered in building responsible generative AI systems.
Model risk functions are not obstacles to generative AI, they are asking the questions any institution should ask before letting a probabilistic system influence customer outcomes. Systems designed with grounding, citation, structured output, and real evaluation answer those questions comfortably. Systems designed as demonstrations do not, and no amount of governance paperwork will rescue them.
Frequently Asked Questions
Why does generative AI struggle in model risk review?
Because validation practice assumes deterministic outputs, stable inputs, and backtestable performance. A language model breaks all three, so the usual evidence cannot be produced in the usual form.
What is the current model risk guidance in the US?
SR 26-2, issued 17 April 2026, replaced SR 11-7 and SR 21-8. Its OCC companion bulletin states the revised guidance explicitly excludes AI and generative AI models, so it is not the framework your generative system is judged against.
Is a generative AI system a model for inventory purposes?
Usually yes where its output informs a business decision. Classify it explicitly rather than avoiding the question, because an unregistered system used in decisions is the finding you least want.
How do you validate a non-deterministic system?
With a held-out evaluation set carrying real ground truth, measured repeatedly across runs, plus human review sampling, adversarial testing, and monitored drift rather than a single accuracy figure.
How does effective challenge work when you cannot inspect the model?
You challenge the system rather than the weights: the use case boundary, the grounding data, the evaluation design, the failure modes, and the human controls around the output.
Why is retrieval grounding a control rather than a feature?
Because it constrains the model to cited source material, which converts an unverifiable assertion into a claim a reviewer can check against a document.
Which use cases pass review most easily?
Ones where a human decides, the source material is controlled, and errors are visible and recoverable, such as drafting, summarising, and retrieval over internal documents.
What belongs in the approval package?
Use case boundary, inventory classification, grounding and data lineage, evaluation methodology and results, failure modes, human controls, monitoring plan, and named owners.
