Back to Executive Insights

Executive Insights

Where Agentic AI Engagements Die Between Pilot and Production

The pilot works. The production deployment does not. The gap is predictable, and nearly all of it traces back to decisions made in the first two weeks.

There is a recognizable shape to agentic AI engagements that fail. The pilot is impressive. The demonstration earns executive attention and a budget. Then production takes three times the estimate, ships in a materially degraded form, or quietly never ships at all.

Six failure modes recur. None of them is a model failure. Each traces to a decision made early, usually before anyone believed they were making an architectural commitment. They also arrive in order: the first three are set during the pilot itself, the last three surface at go-live and after.

The pilot ran on curated context

The demonstration used a clean, hand-selected corpus. Production data is duplicated, contradictory, partially stale, and permission-scoped differently by department. Accuracy that presented as ninety-five percent in the pilot is materially lower against real inputs, and the gap is not a tuning problem. It is a data problem that the pilot was structured to avoid.

The root cause is scoping: the pilot was designed to prove the model could perform the task, not to prove the task survives the client's production data estate. The correction costs almost nothing at the outset — run the pilot against a representative slice rather than a curated one, including the duplication, contradictions, and stale records the estate actually contains, selected by someone with direct knowledge of its condition.

No evaluation harness, so nobody can prove anything

Someone adjusts a prompt, upgrades a model version, adds a tool. Did that help or hurt? Without a golden dataset and a scored regression run, the honest answer is that nobody knows, and the team is reduced to arguing from anecdote and screenshots.

The same discipline extends to change control. If prompts, knowledge sources, and tool definitions are not versioned, and changes are not reversible, a team cannot reproduce a result it liked or roll back a regression it did not notice. The harness measures; versioning is what makes the measurement actionable.

This is the most common reason a system that genuinely works cannot be certified as working. It also makes the go-live decision structurally unmakeable. No serious governance board approves a system whose performance cannot be expressed as a number, and no client accepts a deliverable whose acceptance criteria were never defined. The harness is not a quality nicety. It is the artifact that lets the engagement close.

Identity and authorization deferred to the end

The pilot ran under a service account with broad permissions, because that was the shortest path to a working demonstration. Then security review asks how the agent's actions are attributed to a human principal, whether it can act beyond the requesting user's entitlements, how authority is revoked, and what the audit trail looks like to an examiner.

Retrofitting delegated authority into a system architected around a service account is not a configuration change. It is frequently a redesign of the execution path. In regulated and federal environments this is the single most common hard stop between a successful pilot and a production authorization.

Cost discovered at scale

Pilot volumes are small enough that inference spend is a rounding error. At production volume the same architecture produces a number that changes the business case.

The causes are architectural rather than operational — caching behavior, orchestration design, retry policy, model selection — which is why they are inexpensive to change in week two and structural by month six. But the failure is not that the pilot was expensive. It is that the pilot was never instrumented to produce a cost-per-transaction figure anyone could extrapolate, so the first credible number arrives after the architecture has hardened around it.

Tool surface drift

The agent depends on APIs, schemas, and downstream systems owned by other teams, and those change on their own schedules. A pilot running for six weeks never encounters this. A production system encounters it continuously.

Without contract tests around the tool surface and monitoring that distinguishes agent failure from dependency failure, the system degrades silently. The client's experience is that it worked in June and is unreliable by October, and nobody can say why.

No owner after handoff

The engagement closes, the team demobilizes, and the system enters an environment where models are deprecated, prompts drift out of alignment with changed business processes, and the evaluation set stops being run because running it was never assigned to anyone.

Six months later the client's position is that the system got worse and the integrator's position is that it was working at handoff. Both are accurate. Agentic systems carry an operating discipline, and if the statement of work does not name who holds it, the answer is nobody.

The pattern underneath all six

Every one of these is an engineering discipline that a proof of concept is structured to skip — and skipping them is the correct decision for a proof of concept. The point of a pilot is speed. The failure is not the pilot. The failure is treating the pilot as an architectural head start when it was deliberately built as a shortcut.

The practical correction is a production readiness assessment conducted before scale-up is approved, not after it stalls. Six questions carry most of the weight:

  • What does accuracy look like against representative production data, including its known defects?

  • What is the evaluation harness, who runs it, and what threshold constitutes acceptance?

  • How does the agent obtain authority, whose authority is it, and how is it revoked?

  • What is cost per transaction at one hundred times current volume?

  • What happens when an upstream dependency changes without notice?

  • Who owns this system in month seven, and under what agreement?

An organization that answers those six questions before scaling is not guaranteed a successful deployment. An organization that cannot answer them is very reliably heading toward one of the six conditions above, and usually toward more than one at once.