The AI centre of excellence is a transition state
Centralising capability is the right first move and the wrong steady state. The question worth arguing about is what the CoE is supposed to make itself unnecessary at.
Almost every large organisation reaches for the same structure in the first year: pull the scarce people into one team, give it a name with “excellence” in it, and route all the AI work through it. This is the correct first move. It is also a structure with a natural half-life, and the failure mode is not choosing it — it is forgetting to say out loud when it should end.
Why centralising is right at the start
At the beginning, the binding constraint is judgement, not capacity. Almost nobody in the organisation has shipped one of these systems, so nobody can tell a hard problem from an easy one, a real evaluation from a demo, or a vendor claim from a vendor. Scattering six capable people across six business units means six teams each learning the same lesson slowly and privately.
Centralising does three things well:
- It concentrates reps. One team doing twenty projects learns twenty times. The pattern library — which use cases fail, what second line asks, what the real unit costs look like — only forms above a certain density of experience.
- It gives risk a single counterparty. Second line, legal, procurement and audit each get one relationship to build instead of thirty. In the first year this alone can be worth more than the engineering.
- It makes the paved road buildable. Shared evaluation harnesses, a vetted model gateway, logging that satisfies audit — these are only worth building once there is one team who owns them.
Where it stops working
The trouble starts when demand outruns the team, which happens faster than anyone plans for. Three symptoms show up in order.
First, the CoE becomes a queue. Business units wait months for a slot, and the ones with budget quietly start their own thing. You now have shadow AI and a bottleneck, which is the worst of both structures.
Second, the CoE becomes the domain expert by proxy. Its members are good at the technology and necessarily shallow on the business process being changed. The projects that work are the ones where a business person cared enough to sit with the team every day; the projects that fail are the ones where the CoE was handed a brief and asked to come back with a system. This is not a talent problem. Process knowledge does not transfer through a requirements document.
Third — and this is the one that ends programmes — the CoE owns delivery but not the P&L. It gets measured on systems shipped, not on outcomes changed, because outcomes belong to the business unit. Everyone can see the mismatch and nobody can fix it from inside the structure.
“A centre of excellence that is still the only place excellence happens in year three has failed at its actual job.”
What the second structure looks like
The destination most organisations converge on is a small central platform-and-standards function, with delivery capability embedded in the business units.
The central team keeps what benefits from being singular: the model gateway and its commercial terms, the evaluation and observability tooling, the risk tiering standard and the relationship with second line, the reference patterns, and the community of practice that keeps embedded engineers from re-learning things privately. Roughly: everything that is a road, plus the right to say what “done” means.
The business units keep delivery, because they own the process, the data, and the outcome. Their engineers use the paved road and are held to the standard, but they report to the P&L that changes if the work is good.
What matters is that this is a deliberate handover with a trigger, not a drift. Useful triggers: the CoE’s queue exceeds a set wait; a business unit has run three projects on the paved road without central delivery help; the standards are stable enough that a team can self-certify a tier 3 use case. Write the trigger down in the founding charter of the CoE, in the first month, when it costs nothing to agree.
The transitional question worth asking
Every CoE charter should answer one question explicitly: what are we trying to make unnecessary, and how will we know we have?
If the answer is “nothing — we are the permanent home of AI delivery”, that is a defensible choice for a smaller organisation, but say so, and staff it for the demand it will actually attract.
If the answer is “central delivery”, then the CoE’s own success metric should include things like the number of business-unit engineers shipping on the paved road, the share of projects the CoE never touched, and the time from idea to tier-3 approval without central involvement. A team measured on those numbers behaves very differently from a team measured on systems delivered — it writes documentation, it runs clinics, it resists the flattering request to just build it for them.
The organisations that got this right did not have a better first structure. They had the same one, plus a written expiry date and the discipline to honour it.