Retrieval is not a knowledge base
Teams ship retrieval as though it were search with better manners. It is a data product, and it fails in the ways data products fail — quietly, and mostly on freshness.
Retrieval-augmented generation gets built by application teams and reasoned about as though it were search. Point it at a corpus, tune the chunking, ship it. The demo is excellent, because a demo asks questions the corpus answers well.
What has actually been built is a data product with a language model on the front. It fails the way data products fail, and almost none of those failure modes look like a model problem.
Freshness is the first one
A search index that is a week stale is annoying. A retrieval system that is a week stale is confidently wrong, in fluent prose, with a citation. The generation layer removes every signal a user would normally have that the information is old — no result timestamps, no obvious gaps, no sense of how much was scanned.
So the reindex cadence stops being an operational detail and becomes a correctness property. It belongs in the design, with an owner and an alert, next to the freshness guarantees you would put on any other data product.
The model does not know what it did not retrieve, and neither does the person reading the answer.
Then coverage
The second failure is subtler. A system that retrieves nothing relevant does not usually say so. It answers from whatever it did retrieve, or from parametric memory, and the answer is often plausible enough to pass.
This is why retrieval needs its own evaluation, separate from the end-to-end one. Measure retrieval quality directly — was the passage that contained the answer in the returned set — because an end-to-end score averages that failure away against the questions the corpus happens to cover well.
Permission is the one that ends careers
Every retrieval system inherits the access assumptions of everything it indexes, including the mistakes. If the crawl could read it, the model can surface it, to anyone who asks a question that matches. Permission has to be evaluated at query time against the person asking, not resolved once when the index was built.
That is a straightforward requirement and an expensive retrofit, which is the usual reason it is missing.
Treat it like what it is
None of this is exotic. It is the standard discipline for anything that serves data to people: a named owner, freshness targets you monitor, coverage you measure, access enforced at read time, and a test suite that fails loudly when the corpus shifts underneath you.
The teams that get retrieval right are rarely the ones with the best embedding model. They are the ones who noticed early that they had shipped a data platform, and staffed it accordingly.