Why do so many AI pilots never make it into production?
Because pilots are run under conditions production never provides: a clean data sample, a motivated human watching every output, one happy path, and no owner of the boring parts. Production adds volume, edge cases, expiring credentials, staff who did not attend the kickoff, and a vendor whose API changes without notice. The fix is to pilot on the real data path from day one and to write the promotion criteria before the pilot starts.
The pilot is usually a different system than the one you wanted
A typical pilot exports three months of records to a spreadsheet, runs them through a model, and produces a convincing set of results. Everyone agrees to move forward. Then someone asks how the data will arrive next Tuesday, and the answer is that a person exported it.
That gap is not a detail. The exported file was cleaned, deduplicated and complete. The live feed is late, partial, and full of records nobody expected. The pilot proved the model could do the task; it proved nothing about whether the task could be done continuously.
The failure modes, in rough order of frequency
- No live data path. The pilot ran on an extract. Building the real feed turns out to be most of the project.
- No owner. Output goes to a shared inbox nobody reads. Adoption dies quietly and is later described as an accuracy problem.
- Undefined success. Nobody wrote down what would count as good enough, so the decision defaults to whoever is least enthusiastic.
- Edge cases at volume. The one percent of records with missing fields becomes hundreds per week and erodes trust faster than the good outputs build it.
- Silent breakage. A token expires, a sync stops, and dashboards keep rendering yesterday's numbers as if nothing happened.
What a promotable pilot does differently
Run the pilot through the same connection you intend to run forever, even if it only covers one location, one crew or one campaign. Narrow the scope, never the plumbing. If the API is the risky part, the pilot exists to test the API.
Then write the promotion criteria in advance and in plain language: what metric, measured how, compared against what baseline, decided by whom, on what date. A pilot without a decision date does not end — it just gradually stops being discussed.
Budget for the unglamorous half
Monitoring, retries, backfills, permission renewals and mapping updates are not overhead on top of the project. They are the project's second half, and they are why integration work is engineering rather than configuration.
When a platform absorbs most of that maintenance burden, the remaining custom work gets much smaller. That is the case for starting on a configurable platform and customizing only the part that is genuinely specific to your business.
Topics: pilots · production · failure modes · project risk
Have a version of this question about your own business?
The useful answer usually depends on which systems you run and how they're connected. That's a conversation, not a blog post.