Weak models are rarely the reason AI pilots stall. Most pilots were designed to answer a research question, and production asks an operations question. If you want a pilot to reach production, scope it as the first month of the operation it is meant to become, and agree before it starts what happens when it works.
Most AI pilots never become the way the work gets done
The evidence on this is consistent across very different studies.
- The MIT NANDA report "The GenAI Divide" (July 2025) found that sixty percent of organizations evaluated enterprise-grade AI systems, "but only 20 percent reached pilot stage and just 5 percent reached production."
- IBM's 2025 survey of 2,000 CEOs found that "only 25% of AI initiatives have delivered expected ROI over the last few years, and only 16% have scaled enterprise wide."
- BCG's October 2024 survey of 1,000 executives found that "only 26% of companies have developed the necessary set of capabilities to move beyond proofs of concept and generate tangible value."
Size works against you here. MIT NANDA reports that large enterprises, defined as firms with over $100 million in annual revenue, "lead in pilot count" yet "report the lowest rates of pilot-to-scale conversion." Top mid-market performers reported an average of 90 days from pilot to full implementation; enterprises "took nine months or longer." Large companies, by MIT NANDA's definition, run more pilots and convert fewer of them.
The failure happens around the model
When MIT NANDA's researchers asked why enterprise AI systems fail, they did not land on model quality. "Most fail due to brittle workflows, lack of contextual learning, and misalignment with day-to-day operations." Their list of myths includes the belief that model quality, legal, data and risk are the main brakes; their finding is that "most AI tools don't learn and don't integrate well into workflows."
That matches what an operations leader sees on the ground. The model classifies the sample tickets well. Then the pilot meets the real queue: tickets that reference three accounts, a customer who replied in a different language, a billing system that times out every Friday evening, an agent who does not trust the suggestion and redoes the work anyway. None of those are model problems. All of them decide whether the system can carry real volume.
A pilot scoped as an experiment never meets them. It runs on an export, in a sandbox, with the strange cases filtered out so the results are clean. It passes. Then the organization discovers that connecting it, handling the exceptions and deciding who owns it is a second project with no budget, no sponsor and a team that has already moved on.
An experiment and an operation are scoped differently
| Dimension | Pilot scoped as an experiment | Pilot scoped as an operation |
|---|---|---|
| Data | A cleaned historical export | Live cases from the system of record |
| Volume | A fixed test set | A controlled share of real daily volume |
| Systems | A sandbox or a separate interface | The tools the team already uses |
| Exceptions | Filtered out to keep results clean | Routed to named people from day one |
| Owner | A data or innovation team | The leader who owns the process |
| Success measure | Model accuracy on the test set | Cycle time, cost per case, backlog and error rate against a baseline |
| Running cost | Not estimated | Known before the pilot starts |
| What success triggers | A business case for another budget round | A move to full volume under terms already agreed |
The right-hand column costs more effort in the first weeks. It also removes the second project, and it is the shape the five-step method in the AI implementation guide gives a pilot. When the pilot ends, the system is already in the right place, the exceptions already have owners and the numbers already mean something to the person who signs off.
Size the pilot so that success is the contract
The last row of that table is commercial, and it matters as much as the technical rows. Many pilots die in procurement: a success that needs a new business case, a new vendor review and a new budget cycle loses momentum for months, and by then the sponsor has changed jobs.
The way out is to agree the end state before the start. Define the measures the pilot must hit, the volume it will grow to, and the terms for running it if it does. Then a pass changes the scope of an agreement that already exists. MIT NANDA describes the buyers who cross the divide in these terms: they "act like BPO clients, not SaaS customers," "demand deep customization" and "hold vendors accountable to business metrics."
Manthle sets its pilots up on those terms: a pilot that works keeps running, operated by Manthle's engineers with the client's team, as laid out under AI implementation services. Whoever you work with, ask the question before the pilot starts. What happens on the day it succeeds?
Four signs a pilot is drifting into purgatory
- Nobody in operations can name the success measure. If the pass mark lives in a data science notebook, the business will not act on it.
- The pilot has its own screen. A separate interface means a separate habit, and habits lose to the queue.
- The exceptions are "phase two." They are often where much of today's effort goes, and deferring them defers the decision.
- The running cost is unknown. A system nobody has priced to operate is a system nobody can approve to operate.
What to do with a pilot that is already stuck
Rescope it, using decision gates like the ones in the AI implementation roadmap. Pick one process slice where the pilot showed promise, connect it to the live system, put the exceptions in scope with a named owner, take a baseline, and set a date for a go or stop decision. A pilot that cannot hit operational measures on live volume should end, and the lesson is worth more than another renewal.
If it does hit them, move it to production on the terms you agreed. The step after that, and the one most programs forget, is running the deployed system with the same care that went into building it. A pilot that was scoped as an operation reaches that step with its owner, its exception queue and its numbers already in place, which is the whole point of scoping it that way.