There is a statistic that has followed this track like a shadow, and this is the article where we finally turn around and face it: roughly 88% of AI agent pilots never reach production, and across AI initiatives broadly, only about a quarter deliver their expected ROI and barely 16% achieve enterprise scale. If those numbers described a surgical procedure, nobody would consent to it. Yet organisations keep consenting, quarter after quarter, because each pilot individually feels promising and the mortality happens later, quietly, in the gap between demo and deployment.
Here is the reframe this article is built on: the gap is not a technology problem, and treating it as one is why it persists. When researchers autopsy the failures, model quality barely features. The recurring causes are unclear success criteria, insufficient data and tool access, missing evaluation coverage, and nobody owning the outcome, in other words, decisions that were made (or not made) before the pilot ever started. Pilot mortality is a planning disease with a production diagnosis.
Which is genuinely good news, because planning diseases are curable. This article is the cure as I practise it: the anatomy of the gap, the gates that close it, and the operating habits of the cohort that ships.
The anatomy of the gap
Why do pilots that work stop working on the way to production? Four structural gaps, each predictable, each routinely unplanned-for.
The data gap. The pilot ran on curated data: a clean extract, a hand-picked document set, a friendly week of tickets. Production runs on the real distribution: missing fields, stale documents, adversarial phrasing, the Tuesday-afternoon weirdness no one curated for. Quality drops, and the team misdiagnoses a data gap as a model gap, beginning the vendor-swapping doom loop. Everything in this track’s data arc exists to close this gap before it opens.
The integration gap. The pilot lived beside the workflow: a separate tab, a copy-paste bridge, an enthusiastic human ferrying context. Production must live inside the workflow: real systems, real permissions, real latency budgets, security review, the access paths that take three months to provision. Pilots that defer integration are deferring most of the project and calling the remainder success.
The trust gap. The pilot was used by believers. Production must be used by everyone, including the sceptics whose workflow just changed without their consent. Adoption is not a launch email; it is redesigned processes, training, feedback channels, and visible responsiveness to the first complaints. Systems technically in production but practically ignored are the stealth mode of the 88%.
The economics gap. The pilot cost was a rounding error. Production cost is tokens at volume, infrastructure, monitoring, and the ongoing human attention every AI system needs. Use cases that pencilled at pilot scale can drown at production scale, which is why the sourcing article insisted on modelling production economics before commitment.
Notice the common structure: each gap is invisible in the demo and decisive after it. A pilot, properly understood, is an instrument for exposing these gaps early, which leads to the central discipline.
Design the pilot as a production rehearsal
The single highest-leverage change most organisations can make: stop designing pilots to demonstrate value and start designing them to rehearse production. The distinction sounds subtle and changes everything downstream.
A demonstration pilot optimises for the best case: curated data, friendly users, success theatre at the steering committee. A rehearsal pilot deliberately samples the worst case: real data distribution including the mess, at least one genuine system integration, sceptical users in the cohort, and costs tracked at projected volume. A rehearsal pilot is allowed to fail, indeed is designed to fail early if failure is coming, because a pilot that fails in eight weeks for reasons you now understand is a bargain against a production system that fails in month eleven.
Three artefacts must exist before a rehearsal pilot starts, and their absence is the most reliable predictor of eventual mortality I know.
Written success criteria with baselines. The metric, its current value, the target that would justify production investment, and the measurement method, agreed by the business owner before day one. The plurality of negative-ROI deployments trace to precisely this omission: without pre-committed criteria, every pilot succeeds rhetorically and the production decision gets made on demos and hope.
An evaluation suite. Your cases, your edge cases, your quality thresholds, runnable on demand. This is the same infrastructure the model portfolio and the trust ladder depend on, and the differential it produces (organisations with systematic evaluation move roughly six times more systems into production) makes it the best-evidenced investment in the entire AI operating stack.
A named owner with production authority. Not a pilot sponsor: an owner whose remit explicitly includes taking this to production and running it there, with budget line attached. Pilots owned by innovation teams and destined for handoff die at the handoff with remarkable consistency, because the receiving team inherits a system they never wanted, built on assumptions they never agreed to.
Figure 1 contrasts the two pilot designs and the artefacts the rehearsal design requires up front. The uncomfortable honesty of Figure 1 is that most organisations, shown the two columns, recognise their own pilots in the left one.
The three gates
Between rehearsal and scale, I recommend three explicit gates, each a real decision with kill authority. Figure 2 shows the full pipeline; the gates are its joints.
Gate one: the evidence gate, at pilot end. Did the metric move against baseline per the pre-committed criteria, on real data, at rehearsed conditions? The three honest outcomes are proceed, iterate (once, with a specific hypothesis), or kill. Killing at gate one is a system success, not a failure, and how leadership treats the first killed pilot teaches the organisation whether the gates are real. Celebrate it accordingly: publicly, with the lessons written down.
Gate two: the readiness gate, before rollout. The production checklist, none of it optional: data pipelines with monitoring and owners (not the pilot’s extract), security and governance sign-off through the appropriate lane, integration complete, evaluation suite wired into deployment so regressions surface before users find them, action logging where the system acts, unit economics validated at volume, and the operational runbook: who gets paged, what the rollback is, how users report problems. This gate is where the four gaps get formally closed, and rushing it is how they reopen in production.
Gate three: the value gate, after controlled rollout. Roll out to a bounded population first (one region, one team, one queue), measure against the same criteria at production conditions, then decide scale, hold, or roll back. The bounded rollout is the cheapest insurance in the pipeline: most of what production teaches, it teaches at 10% scale for 10% of the cost.
The pipeline in Figure 2 looks slower than the enthusiasm path, and per initiative it is, by perhaps six weeks. Across the portfolio it is enormously faster, because the enthusiasm path’s speed is an illusion created by not counting the eleven dead pilots behind each survivor.
The operating habits of the shipping cohort
Beyond the gates, the organisations that consistently cross the gap share operating habits worth stealing.
They run fewer pilots, deliberately: portfolio slots are scarce, so every pilot that starts has cleared the value-feasibility bar and carries its three artefacts. They reuse rails: each production crossing leaves behind data products, integration patterns, evaluation harnesses, and governance precedents that the next crossing inherits, which is why their second year is so much faster than their first. They keep humans in the economics: production plans include the ongoing attention (evaluation review, feedback triage, drift response) as a budgeted cost, not a surprise. And they publish their funerals: killed pilots get a one-page post-mortem in a shared register, converting each death into portfolio-wide calibration.
None of this is glamorous, and that is rather the point. The gap between pilot and production is where AI strategy stops being ideas and becomes operations, and operations reward discipline over brilliance every single time. It is also, encouragingly, where the compounding hides: an organisation that crosses the gap once with the full apparatus (artefacts, gates, rails, funerals) crosses it the second time in half the effort, and by the fourth crossing the apparatus has become simply how things ship. The 88% is not a law of nature. It is the aggregate cost of skipping steps, paid by organisations that mistook the steps for bureaucracy.
What remains for this track is assembly: turning the portfolio, the sourcing posture, the agent strategy, and this pipeline into a sequenced, fundable twelve-month plan, and then a final tour of the anti-patterns that undo good strategies. The roadmap is next.