Why AI pilots die in model risk review
SR 11-7 wasn't written for systems that change weekly. Here's the tiered validation pattern that gets agents through second-line review.
Every bank I have worked in has a graveyard of successful pilots. The demo worked. The metrics were real. The sponsor was genuinely enthusiastic. And eighteen months later the system serves no traffic, because the questions that killed it were never on the pilot’s scorecard.1
The usual post-mortem blames the second line — risk was too slow, too conservative, too attached to a framework written for credit scorecards. That explanation is comfortable and wrong. The gate is doing what it was designed to do. The problem is that we keep presenting it with an object it cannot evaluate.
The pilot theater problem
A pilot optimises for the demo. It runs on a curated slice of data, in a sandbox, with the people who built it watching. Its scorecard measures whether the output looks right.
Production optimises for the run. It runs on everything, unattended, for years, and its scorecard measures whether anyone can explain what happened on a Tuesday in March when the output was wrong and a customer was harmed.
These are not the same test, and passing the first tells you very little about the second. Worse, a successful pilot creates a specific kind of organisational damage: it convinces everyone the hard part is done. Budget gets released for the wrong phase. The team that holds all the context disbands. The system arrives at the risk gate as an orphan, sponsored by nobody who can answer a question about it.
What second line is actually asking
Model risk management in US banking runs on SR 11-7, the Fed and OCC’s 2011 supervisory guidance. It is a better document than its reputation suggests. Strip away the vocabulary and it asks three things:
- What is this thing, and what is it for? A documented statement of purpose and intended use, so that use outside that envelope can be identified as a breach rather than a surprise.
- How do you know it works? Evidence, generated independently of the people who built it, that the system performs as claimed across the range of conditions it will actually meet.
- Who is accountable when it doesn’t? A named owner, an escalation path, and a control that fires before the harm rather than after.
None of that is hostile to AI systems. All of it assumes a model is a fixed artifact — a set of weights and a scoring function you can validate once, revalidate annually, and version when it changes.
An agentic system is not a fixed artifact. The weights move when the vendor ships. The prompt moves when someone improves a phrasing. The tool surface moves when an API adds a field. The retrieval corpus moves continuously, by design. Validation-on-the-instance means revalidating weekly, which no second line can staff, so the request queues, and while it queues the instance changes again.
“The gate isn’t too strict. It’s aimed at the wrong object.”
Validate the envelope, not the instance
The pattern that works is to move the unit of validation up a level. You stop asking second line to approve this system as it is today and start asking them to approve the envelope this system is permitted to move within, and the controls that detect when it leaves.
Concretely, the artifact you submit is not a model card for one configuration. It is four things:
- A bounded statement of use. What decisions this system may influence, what it may never do unaided, which populations it touches. Narrow beats broad — a narrow envelope approved this quarter is worth more than a wide one still in review next year.
- The change envelope. Which components may change without re-review, within what tolerances, and what change automatically forces re-review. A prompt edit inside a tested template is routine; adding a tool that can move money is not.
- Continuous evidence. An eval suite that runs on every change and reports against the same measures used at approval, with results second line can read without asking you. This is the load-bearing piece and the one most teams skip.
- The failure path. What the control is, what it fires on, who receives it, and what they are empowered to do. “The model has a confidence threshold” is not a control. “Below threshold routes to a named queue with a four-hour SLA and a monthly sampling review” is.
The reframe is not a trick to get past the gate. It genuinely gives the second line something more useful than an instance snapshot — a snapshot goes stale the day after it is signed, while an envelope with live evidence stays true.
Tiering, so the effort lands where the risk is
Applying that machinery uniformly is how you end up with an eight-month queue for a meeting-notes summariser. Tier by consequence, not by technical sophistication:
| Tier | Test | Treatment |
|---|---|---|
| 3 | No external effect; a human reads every output | Register it, log usage, review annually |
| 2 | Influences a decision about a customer, with a human in the loop | Full envelope, independent eval, quarterly control testing |
| 1 | Acts unaided, or the harm is not reversible by the human | Tier 2 plus pre-deployment challenge, kill switch, standing committee ownership |
The tiering test that matters is not “how clever is the model” but “if this is wrong and nobody notices for a week, what is the worst outcome and can we undo it?” A regex can be tier 1. A frontier model can be tier 3.
Getting this table agreed with the second line before you build anything is the single highest-leverage hour available to an AI programme. It converts an unbounded conversation about AI safety into a routing decision, and routing decisions can be made in days.
What to fix before the next pilot
Three things, none of which are technical:
Name the production owner on day one. Not the pilot sponsor — the executive who will hold the run-rate, the incidents, and the annual attestation. If nobody will take it, that is the finding, and you have learned it for the cost of a meeting rather than a programme.
Fund the run, not the build. Innovation budgets end at the demo. Ask for the three-year operating line — evals, monitoring, revalidation, second-line time — in the same paper that asks for the build. A pilot that cannot survive that question was never going to reach production.
Bring second line in at design, not at the gate. They are not the last approval step; they are a design constraint you can either discover early and cheaply or late and expensively. Every risk officer I have worked with would rather shape a system in February than reject it in November.
None of this makes governance free. It makes it sequenced, which is a different and achievable thing. The organisations getting agents into production are not the ones with the most permissive risk functions. They are the ones that stopped treating the risk function as a thing that happens to them at the end.
- Fed SR 11-7, "Guidance on Model Risk Management," 2011 — the reference regime for US banks, and the direct ancestor of most of what your second line will ask for.