Chris on AI

Evaluations are not tests

Chris Taylor — Tue, 04 Aug 2026 00:00:00 GMT

Engineering teams arrive at AI systems fluent in testing, and reasonably assume evaluation is testing with fuzzier assertions. The vocabulary encourages it: both run in CI, both go green or red, both are described as coverage. The assumption is wrong in a way that takes about six months to become obvious, usually when a suite that has been green throughout ships a regression everyone can see in the product.

The difference that matters

A test asks a closed question with a knowable answer: given this input, does the function return this output? It is deterministic, it is cheap, and a passing test is a proof. When a test fails, something is broken.

An evaluation asks an open question with no fixed answer: across this distribution of inputs, is the system’s behaviour good enough for what we are using it for? It is statistical, it costs real money, and a passing eval is evidence, not proof. When an eval score drops, something may be broken, or the sample may have shifted, or the judge may have drifted, or the threshold may have been wrong.

Three consequences follow, and each one contradicts a testing habit that is otherwise a virtue.

A single run is a sample, not a result. One green run means little. What you need is a distribution and a trend — the same suite, run repeatedly, tracked over releases. Teams that treat the first green as pass/fail throw away the only signal that actually predicts regressions.

100% is a warning sign. In testing, a suite that always passes is a suite doing its job. In evaluation, it means the cases are too easy, and the suite has stopped being able to detect the thing you built it for. Healthy suites sit somewhere uncomfortable and move.

The suite is a living asset, not a ratchet. Test suites accumulate — you rarely delete a test. Eval sets need active curation: retiring cases the system has fully absorbed, and continuously adding the failures found in production. An eval set that has not changed in two quarters is measuring last year’s system.

“A suite that always passes has stopped being a measurement and become a ritual.”

What earns its place in a suite

The useful mental model is three layers, and they are not interchangeable.

The middle layer gets the attention and the budget. The two either side of it catch what it cannot.

Assertions. The genuinely deterministic properties: valid JSON, schema conformance, no PII in the output, required disclaimer present, tool called with well-formed arguments. These are tests, they are cheap, and they belong in CI on every commit. Most teams under-invest here and over-invest in the layer above, because assertions feel too simple to be the answer to an AI problem. A surprising share of production incidents are assertion failures.

Graded cases. A curated set with known-good answers, scored by a rubric — sometimes by a model, sometimes by an exact or fuzzy match. This is the layer that tracks quality over time. It needs enough cases to be statistically meaningful, and it needs to be built out of real traffic rather than invented examples, because invented examples encode your assumptions about what users do.

Human review. A sampled slice looked at by someone who knows the domain. Expensive, slow, and the only layer that can find a failure mode you did not think to encode. It is also the layer that calibrates the one above it: if your model-as-judge disagrees with your domain expert, the judge is wrong until proven otherwise.

The common mistake is trying to push everything down into the graded layer because it is the automatable one. Assertions catch the cheap failures for almost nothing. Human review catches the novel ones. The middle layer is where you watch the trend, not where you find surprises.

Judges need evaluating too

Model-as-judge is what makes graded evaluation affordable at scale, and it introduces a measurement instrument that can itself drift. Treated carelessly it converts an unknown quality problem into an unknown measurement problem, which is worse because it looks like progress.

Two disciplines make it trustworthy. Hold out a set of human-labelled cases and periodically measure agreement between your judge and your humans — if that agreement number is not tracked, your quality numbers mean nothing in particular. And pin the judge’s model version and prompt, treating a change to either as a change to the instrument: re-baseline, and never compare scores across the boundary. A silent judge upgrade will move every number in your suite and look exactly like a quality change.

Where the suite should live

The organisational placement matters as much as the design. Evals built inside the delivery team and never seen by anyone else are useful for iteration and worthless for assurance — they are marking their own homework, which second line will spot immediately and correctly.

The pattern that holds up: the delivery team owns the eval suite and runs it on every change; the standard for what a suite must contain at each risk tier is owned centrally; and the results land somewhere risk and audit can read without asking. That last part is what converts a development tool into a control, and it is almost entirely a plumbing problem — a durable location, a stable format, and results that are not manually curated before anyone sees them.

Get that right and the eval suite stops being the thing you run before a release. It becomes the continuous evidence that the system is still operating inside the envelope you were approved for — which is the only form of assurance that survives a system that changes weekly.

Why AI pilots die in model risk review

Chris Taylor — Tue, 14 Jul 2026 00:00:00 GMT

Every bank I have worked in has a graveyard of successful pilots. The demo worked. The metrics were real. The sponsor was genuinely enthusiastic. And eighteen months later the system serves no traffic, because the questions that killed it were never on the pilot’s scorecard.¹

The usual post-mortem blames the second line — risk was too slow, too conservative, too attached to a framework written for credit scorecards. That explanation is comfortable and wrong. The gate is doing what it was designed to do. The problem is that we keep presenting it with an object it cannot evaluate.

The pilot theater problem

A pilot optimises for the demo. It runs on a curated slice of data, in a sandbox, with the people who built it watching. Its scorecard measures whether the output looks right.

Production optimises for the run. It runs on everything, unattended, for years, and its scorecard measures whether anyone can explain what happened on a Tuesday in March when the output was wrong and a customer was harmed.

These are not the same test, and passing the first tells you very little about the second. Worse, a successful pilot creates a specific kind of organisational damage: it convinces everyone the hard part is done. Budget gets released for the wrong phase. The team that holds all the context disbands. The system arrives at the risk gate as an orphan, sponsored by nobody who can answer a question about it.

What second line is actually asking

Model risk management in US banking runs on SR 11-7, the Fed and OCC’s 2011 supervisory guidance. It is a better document than its reputation suggests. Strip away the vocabulary and it asks three things:

What is this thing, and what is it for? A documented statement of purpose and intended use, so that use outside that envelope can be identified as a breach rather than a surprise.
How do you know it works? Evidence, generated independently of the people who built it, that the system performs as claimed across the range of conditions it will actually meet.
Who is accountable when it doesn’t? A named owner, an escalation path, and a control that fires before the harm rather than after.

None of that is hostile to AI systems. All of it assumes a model is a fixed artifact — a set of weights and a scoring function you can validate once, revalidate annually, and version when it changes.

An agentic system is not a fixed artifact. The weights move when the vendor ships. The prompt moves when someone improves a phrasing. The tool surface moves when an API adds a field. The retrieval corpus moves continuously, by design. Validation-on-the-instance means revalidating weekly, which no second line can staff, so the request queues, and while it queues the instance changes again.

“The gate isn’t too strict. It’s aimed at the wrong object.”

Validate the envelope, not the instance

The pattern that works is to move the unit of validation up a level. You stop asking second line to approve this system as it is today and start asking them to approve the envelope this system is permitted to move within, and the controls that detect when it leaves.

Concretely, the artifact you submit is not a model card for one configuration. It is four things:

A bounded statement of use. What decisions this system may influence, what it may never do unaided, which populations it touches. Narrow beats broad — a narrow envelope approved this quarter is worth more than a wide one still in review next year.
The change envelope. Which components may change without re-review, within what tolerances, and what change automatically forces re-review. A prompt edit inside a tested template is routine; adding a tool that can move money is not.
Continuous evidence. An eval suite that runs on every change and reports against the same measures used at approval, with results second line can read without asking you. This is the load-bearing piece and the one most teams skip.
The failure path. What the control is, what it fires on, who receives it, and what they are empowered to do. “The model has a confidence threshold” is not a control. “Below threshold routes to a named queue with a four-hour SLA and a monthly sampling review” is.

The reframe is not a trick to get past the gate. It genuinely gives the second line something more useful than an instance snapshot — a snapshot goes stale the day after it is signed, while an envelope with live evidence stays true.

Tiering, so the effort lands where the risk is

Applying that machinery uniformly is how you end up with an eight-month queue for a meeting-notes summariser. Tier by consequence, not by technical sophistication:

Tier	Test	Treatment
3	No external effect; a human reads every output	Register it, log usage, review annually
2	Influences a decision about a customer, with a human in the loop	Full envelope, independent eval, quarterly control testing
1	Acts unaided, or the harm is not reversible by the human	Tier 2 plus pre-deployment challenge, kill switch, standing committee ownership

The tiering test that matters is not “how clever is the model” but “if this is wrong and nobody notices for a week, what is the worst outcome and can we undo it?” A regex can be tier 1. A frontier model can be tier 3.

Getting this table agreed with the second line before you build anything is the single highest-leverage hour available to an AI programme. It converts an unbounded conversation about AI safety into a routing decision, and routing decisions can be made in days.

What to fix before the next pilot

Three things, none of which are technical:

Name the production owner on day one. Not the pilot sponsor — the executive who will hold the run-rate, the incidents, and the annual attestation. If nobody will take it, that is the finding, and you have learned it for the cost of a meeting rather than a programme.

Fund the run, not the build. Innovation budgets end at the demo. Ask for the three-year operating line — evals, monitoring, revalidation, second-line time — in the same paper that asks for the build. A pilot that cannot survive that question was never going to reach production.

Bring second line in at design, not at the gate. They are not the last approval step; they are a design constraint you can either discover early and cheaply or late and expensively. Every risk officer I have worked with would rather shape a system in February than reject it in November.

None of this makes governance free. It makes it sequenced, which is a different and achievable thing. The organisations getting agents into production are not the ones with the most permissive risk functions. They are the ones that stopped treating the risk function as a thing that happens to them at the end.

Fed SR 11-7, "Guidance on Model Risk Management," 2011 — the reference regime for US banks, and the direct ancestor of most of what your second line will ask for.

The AI skills gap is a job-design problem

Chris Taylor — Thu, 18 Jun 2026 00:00:00 GMT

Every firm I speak to describes the same shortage and reaches for the same remedy. We cannot find the modelling talent. So the budget goes to a recruiter, six months pass, three people land, and the shortage has not moved. It has usually got slightly worse, because the three new people now have opinions about a platform nobody owns.

The shortage is real. It is just not the binding constraint.

What the gap actually is

Watch where an AI programme stalls and it is almost never at the point where somebody has to train a model. It stalls where a system that works has to be handed to people who will run it for the next five years — with an on-call rotation, a revalidation cycle, a cost line, and an executive whose name is on the attestation.

That handover needs a person who can hold three things at once: enough engineering to read the pipeline, enough risk literacy to talk to second line without a translator, and enough organisational standing to say no. Firms have those people. They are just not in a job that lets them do it, because that job has not been designed.

The scarce skill is not building the model. It is being accountable for one in production, in a regulated firm, on a Tuesday, when it is wrong.

Why hiring does not fix it

A specialist hire arrives into an org chart that was drawn before the problem existed. If the operating model has not settled who owns an AI system after the pilot team disbands, the new hire inherits the ambiguity rather than resolving it — and the more senior they are, the more expensive that ambiguity gets.

This is the same failure the centre-of-excellence argument runs into from the other side. Capability without an owner does not compound. It pools in whoever happens to care, and it leaves when they do.

Designing the job instead

The fix is unglamorous and mostly free. Write the role down. Name the accountable owner for each production system, and make the name a person rather than a committee. Put the run costs — evaluation, monitoring, revalidation, second-line time — into that owner’s budget, not into the innovation fund that paid for the build.

Then, and only then, work out what you cannot staff internally and hire for that. In most firms the answer turns out to be a much shorter list than the one the recruiter was given.

The test

If you want to know whether you have a skills gap or a job-design gap, ask who gets paged when a production AI system starts behaving differently, and then ask that person what authority they have to stop it. If the two answers are different people, hiring will not help you. You do not need more talent. You need a job that the talent you already have is allowed to do.

The AI centre of excellence is a transition state

Chris Taylor — Thu, 21 May 2026 00:00:00 GMT

Almost every large organisation reaches for the same structure in the first year: pull the scarce people into one team, give it a name with “excellence” in it, and route all the AI work through it. This is the correct first move. It is also a structure with a natural half-life, and the failure mode is not choosing it — it is forgetting to say out loud when it should end.

Why centralising is right at the start

At the beginning, the binding constraint is judgement, not capacity. Almost nobody in the organisation has shipped one of these systems, so nobody can tell a hard problem from an easy one, a real evaluation from a demo, or a vendor claim from a vendor. Scattering six capable people across six business units means six teams each learning the same lesson slowly and privately.

Centralising does three things well:

It concentrates reps. One team doing twenty projects learns twenty times. The pattern library — which use cases fail, what second line asks, what the real unit costs look like — only forms above a certain density of experience.
It gives risk a single counterparty. Second line, legal, procurement and audit each get one relationship to build instead of thirty. In the first year this alone can be worth more than the engineering.
It makes the paved road buildable. Shared evaluation harnesses, a vetted model gateway, logging that satisfies audit — these are only worth building once there is one team who owns them.

Where it stops working

The trouble starts when demand outruns the team, which happens faster than anyone plans for. Three symptoms show up in order.

First, the CoE becomes a queue. Business units wait months for a slot, and the ones with budget quietly start their own thing. You now have shadow AI and a bottleneck, which is the worst of both structures.

Second, the CoE becomes the domain expert by proxy. Its members are good at the technology and necessarily shallow on the business process being changed. The projects that work are the ones where a business person cared enough to sit with the team every day; the projects that fail are the ones where the CoE was handed a brief and asked to come back with a system. This is not a talent problem. Process knowledge does not transfer through a requirements document.

Third — and this is the one that ends programmes — the CoE owns delivery but not the P&L. It gets measured on systems shipped, not on outcomes changed, because outcomes belong to the business unit. Everyone can see the mismatch and nobody can fix it from inside the structure.

“A centre of excellence that is still the only place excellence happens in year three has failed at its actual job.”

What the second structure looks like

The destination most organisations converge on is a small central platform-and-standards function, with delivery capability embedded in the business units.

The central team keeps what benefits from being singular: the model gateway and its commercial terms, the evaluation and observability tooling, the risk tiering standard and the relationship with second line, the reference patterns, and the community of practice that keeps embedded engineers from re-learning things privately. Roughly: everything that is a road, plus the right to say what “done” means.

The business units keep delivery, because they own the process, the data, and the outcome. Their engineers use the paved road and are held to the standard, but they report to the P&L that changes if the work is good.

What matters is that this is a deliberate handover with a trigger, not a drift. Useful triggers: the CoE’s queue exceeds a set wait; a business unit has run three projects on the paved road without central delivery help; the standards are stable enough that a team can self-certify a tier 3 use case. Write the trigger down in the founding charter of the CoE, in the first month, when it costs nothing to agree.

The transitional question worth asking

Every CoE charter should answer one question explicitly: what are we trying to make unnecessary, and how will we know we have?

If the answer is “nothing — we are the permanent home of AI delivery”, that is a defensible choice for a smaller organisation, but say so, and staff it for the demand it will actually attract.

If the answer is “central delivery”, then the CoE’s own success metric should include things like the number of business-unit engineers shipping on the paved road, the share of projects the CoE never touched, and the time from idea to tier-3 approval without central involvement. A team measured on those numbers behaves very differently from a team measured on systems delivered — it writes documentation, it runs clinics, it resists the flattering request to just build it for them.

The organisations that got this right did not have a better first structure. They had the same one, plus a written expiry date and the discipline to honour it.

Your model is only as auditable as its lineage

Chris Taylor — Thu, 02 Apr 2026 00:00:00 GMT

There is a moment in every model risk review where somebody asks what the system was trained on, or retrieves from, or was last evaluated against. It is a reasonable question. It is also, in most firms, unanswerable — not because the answer is bad, but because nobody wrote it down while it was still cheap to write down.

Lineage is a build-time artefact

Data lineage cannot be reconstructed. You can approximate it, and teams do: someone spends three weeks tracing tables backwards through a warehouse, produces a diagram, and everyone agrees it is roughly right. Roughly right is worth very little to a reviewer whose job is to attest to it.

The alternative costs almost nothing if you do it at the time. Every dataset that enters a model carries its source, its extraction time, its permission basis and its version. Every run records the exact versions it consumed. None of that is difficult. It is simply invisible while things are going well, which is why it keeps getting deferred.

Nobody has ever been asked for lineage on a day when everything was fine.

The permission problem underneath it

The harder version of this question is not where the data came from but whether you were allowed to use it that way. A retrieval system that indexes an internal share drive inherits every access assumption that drive ever made, including the wrong ones. The model does not know that the folder was readable because of a 2019 migration error. It will cheerfully summarise it for whoever asks.

This is where lineage stops being a documentation exercise and becomes a control. If you can trace a retrieved passage back to a document and that document back to a permission set, you can enforce something. If you cannot, your access model is whatever the index happened to crawl.

What good looks like

The bar is lower than people assume. Three properties cover most of it:

Every artefact — dataset, index, model version, eval run — has an identifier that appears in the logs.
Every production inference can be traced to the artefact versions that produced it, without asking a person.
Permission is evaluated at query time against the requesting user, not baked in when the index was built.

None of that requires a platform purchase. It requires deciding, before the first system ships, that the answer to “what is this built on” should be a query rather than an investigation.

The cost of deferring

The reason this is worth arguing about now rather than later is that lineage debt compounds in a way most technical debt does not. An undocumented dataset gets copied, joined, and derived from. Three quarters later the thing you cannot explain is not one table but a small ecosystem, and the reconstruction cost has gone from a week to a project nobody will fund.

The systems that clear review are rarely the ones with the best models. They are the ones that can answer questions about themselves.

Your cloud bill is a design review you keep skipping

Chris Taylor — Mon, 09 Mar 2026 00:00:00 GMT

The monthly cloud bill lands with finance, gets converted into a variance figure, and reaches engineering as a request to reduce spend by some percentage. By then the information has been stripped of everything that made it actionable. What was a specific statement — this retry loop calls a cross-region endpoint on every failure — arrives as a target.

The bill is not a finance artifact that engineering must occasionally react to. It is the most complete record you have of how your architecture actually behaves under real load, and almost nobody reads it that way.

What the bill knows that your diagrams don’t

An architecture diagram records intent. The bill records what happened. Where they disagree, the bill is right.

Three things show up there and essentially nowhere else:

The paths you did not know were hot. Egress charges between availability zones are the classic tell. Nobody designs a chatty cross-AZ call; it emerges from a service that got split, a client library defaulting to a regional endpoint, or a retry policy that fans out. The diagram shows one arrow. The bill shows how many times it was traversed.
The idle you are paying to keep warm. Non-production environments running at production shape, over-provisioned baselines sized for a launch that already happened, storage tiers holding data nobody has read in a year. None of this fails, so nothing alerts.
Your real unit economics. Cost per tenant, per transaction, per inference. Most teams cannot produce this number, which means they cannot tell whether growth improves their margin or erodes it — and that is a strategy question, not a housekeeping one.

The three questions that recover most of it

Before any tooling, the same three questions do most of the work:

Where does the data move, and how often? Compute is visible and gets attention. Data movement is invisible in the diagram and frequently the largest line. Trace the ten most common request paths and count region and zone boundaries crossed.

What is running that nobody asked for? Not “unused resources” — that framing sends people hunting for orphaned volumes worth a rounding error. The question is which deliberately provisioned things are sized for a condition that no longer holds.

What does one unit of the business cost to serve? Pick the unit that matters — a customer, an order, a document processed — and get to a defensible number. Being roughly right here beats being precisely right about anything else on this list.

“Every recurring cost is a design decision that is still being paid for.”

Where this bites hardest with AI workloads

AI systems break the assumptions most cost governance was built on, in three ways.

Cost moves with usage, not with deployment. A traditional service costs roughly the same whether it serves ten requests or ten thousand, because you provisioned it either way. A token-metered system does not. This is genuinely better — spend follows value — but it means an unbounded retry, a runaway agent loop, or a well-meaning batch job can multiply the bill overnight with no infrastructure change to point at.

The expensive choice is usually invisible in the code. Whether a step uses the frontier model or a small one, whether you retrieve five chunks or fifty, whether the agent gets three tool-call turns or twenty — none of these look like cost decisions when written. They look like quality decisions. They are both, and the tradeoff is only legible if someone has put the numbers next to each other.

Evaluation is a real and unbudgeted line item. Running a serious eval suite on every change costs money proportional to how seriously you take it. Teams that did not budget for this quietly stop running the suite, which is a governance failure that first appears as a cost decision.

Making it a review, not a report

The fix is structural and unglamorous: put the bill in front of the people who can change it, at a cadence where changing it is still cheap.

Attribute cost to the team that causes it, not the account that hosts it. Tag enforcement is tedious and it is the precondition for everything else — unattributed spend belongs to nobody and therefore gets optimised by nobody.

Give engineering the shape, not the total. A percentage target produces theatre. “This endpoint is now 40% of the service’s cost and its call volume tripled after the March release” produces a fix, usually within the week.

Put unit cost in the same review as latency and error rate. If cost per transaction lives in a finance deck and p99 lives in an engineering dashboard, the tradeoff between them never gets made explicitly — it gets made accidentally, by whoever last tuned something.

Set a cost budget at design time for anything metered by usage. Not a limit to enforce, a number to be wrong about early. A team that has estimated cost per request before building will notice a 10× surprise in week one rather than in the quarterly variance report.

The goal is not a cheaper bill. It is a bill that nobody is surprised by — which, in practice, is the same thing arrived at from a more durable direction.

The business case that survives second line

Chris Taylor — Wed, 11 Feb 2026 00:00:00 GMT

The standard AI business case has three lines: a headcount-hours figure, an adoption curve, and a licence cost. It clears the investment committee comfortably. It then meets the first serious risk review and loses roughly forty per cent of its value in an afternoon, because the controls the system actually needs were never in the model.

This is not a failure of optimism. It is a failure of scope.

What gets left out

The build is the cheap part, and everyone knows it. What does not appear in the case is the run: the evaluation suite and the person who maintains it, the monitoring, the annual revalidation, the second-line hours consumed by review, the incident process, the retention and the storage that the retention implies.

Individually none of these is large. Together they are frequently the majority of the five-year cost, and they arrive as a surprise because the innovation budget that funded the build has no line for any of them.

A business case that ends at go-live is not a business case. It is a construction estimate.

The tiering shortcut

The reason this matters commercially rather than just procedurally is that control cost is not uniform — it scales with risk tier. A use case that can be honestly tiered low carries a fraction of the assurance overhead of one that cannot.

Which means the single highest-leverage move in most business cases is not squeezing the licence cost. It is designing the use case so that it sits in a lower tier: keeping a human decision in the loop where the decision is consequential, narrowing scope so the failure mode is recoverable, choosing not to touch the data that drags the whole thing into a higher regime.

That is an architecture decision with a price attached, and it is almost always made before anyone thinks to price it.

Writing the case backwards

The version that survives is built in the opposite order. Start from the tier the use case will land in. Take the control set that tier requires as a given cost, not a negotiation. Add the run-rate for the whole life of the system, with a named owner and a budget line. Then see what value is left.

Sometimes the answer is that the case does not work — and finding that out in a planning cycle is enormously cheaper than finding it out after eighteen months of build. More often the case still works, but the shape has changed: smaller scope, tighter boundary, a slower rollout that clears review rather than a fast one that stalls in it.

The number to lead with

If you present one figure, make it total cost of ownership over five years including assurance, against benefit that has been discounted for adoption you can evidence. It is a less exciting number than the one on the first slide. It is also the only one that will still be true in a year, which is the entire point of putting it in front of people who have to fund it.

Retrieval is not a knowledge base

Chris Taylor — Wed, 03 Dec 2025 00:00:00 GMT

Retrieval-augmented generation gets built by application teams and reasoned about as though it were search. Point it at a corpus, tune the chunking, ship it. The demo is excellent, because a demo asks questions the corpus answers well.

What has actually been built is a data product with a language model on the front. It fails the way data products fail, and almost none of those failure modes look like a model problem.

Freshness is the first one

A search index that is a week stale is annoying. A retrieval system that is a week stale is confidently wrong, in fluent prose, with a citation. The generation layer removes every signal a user would normally have that the information is old — no result timestamps, no obvious gaps, no sense of how much was scanned.

So the reindex cadence stops being an operational detail and becomes a correctness property. It belongs in the design, with an owner and an alert, next to the freshness guarantees you would put on any other data product.

The model does not know what it did not retrieve, and neither does the person reading the answer.

Then coverage

The second failure is subtler. A system that retrieves nothing relevant does not usually say so. It answers from whatever it did retrieve, or from parametric memory, and the answer is often plausible enough to pass.

This is why retrieval needs its own evaluation, separate from the end-to-end one. Measure retrieval quality directly — was the passage that contained the answer in the returned set — because an end-to-end score averages that failure away against the questions the corpus happens to cover well.

Permission is the one that ends careers

Every retrieval system inherits the access assumptions of everything it indexes, including the mistakes. If the crawl could read it, the model can surface it, to anyone who asks a question that matches. Permission has to be evaluated at query time against the person asking, not resolved once when the index was built.

That is a straightforward requirement and an expensive retrofit, which is the usual reason it is missing.

Treat it like what it is

None of this is exotic. It is the standard discipline for anything that serves data to people: a named owner, freshness targets you monitor, coverage you measure, access enforced at read time, and a test suite that fails loudly when the corpus shifts underneath you.

The teams that get retrieval right are rarely the ones with the best embedding model. They are the ones who noticed early that they had shipped a data platform, and staffed it accordingly.