What should a first AI pilot actually look like?
A good pilot has a falsifiable question, a baseline measured before it starts, the real data connection it will use in production, a single named owner, and a decision date. Narrow the scope — one location, one channel, one report — but never narrow the plumbing. A pilot that runs on a hand-exported spreadsheet proves the model works and proves nothing about whether the system can exist.
A pilot is an experiment, not a demo
Demos are designed to impress. Experiments are designed to be capable of failing. If there is no result that would cause you to stop, you are not running a pilot — you are running a slow procurement process with extra steps.
State the question as something that can come out false. "Does automated review of every inbound call surface more missed booking opportunities than our current manual sampling?" is falsifiable. "Will AI help our call center?" is not.
Be equally clear about what the pilot is not testing. If the goal is to learn whether the output changes behavior, do not simultaneously use it to evaluate three model vendors. Two variables in one experiment produce a result nobody can interpret afterward.
Measure the baseline before you start
This is the step most often skipped, and it is unrecoverable afterward. Whatever the pilot is meant to improve, write down its current value first: current booking rate, current time spent building the weekly report, current number of calls reviewed per manager per week.
Without that number, the pilot's outcome becomes a matter of impressions. With it, the conversation at the decision date is short. This is why any serious call analysis engagement starts by quantifying what is being reviewed today.
Scope narrow, connect fully
- Narrow the population. One branch, one service line, one campaign. Small enough that a human can spot-check the outputs.
- Keep the real connection. Same API, same schedule, same failure handling you intend to run forever.
- Include an ugly slice. Deliberately include the messy segment — the location with bad data hygiene — because that is where production will break.
- Ship to a real recipient. Output goes to the person who would act on it, in the channel they already use, not to a project folder.
Define what promotion means, in advance
Write down three thresholds before the pilot runs: what result means expand, what means adjust and rerun, and what means stop. Name who decides and on what date. Agreeing to this while everyone is optimistic is much easier than agreeing to it while results are ambiguous.
Also decide who owns the system after promotion. A pilot with no post-launch owner tends to become an orphaned report, which is a more expensive outcome than an honest stop. Where a productized platform already handles the operational upkeep, that ownership burden is a good deal smaller.
Topics: pilot · proof of concept · measurement · project design
Have a version of this question about your own business?
The useful answer usually depends on which systems you run and how they're connected. That's a conversation, not a blog post.