How do we know an operational AI system is actually working?
Measure the decision, not the model. Before launch, record the baseline for the thing you expect to change: how long unbooked calls sit before a callback, how many estimates age out untouched, how quickly a spend anomaly gets caught. After launch, measure the same thing the same way. Track adoption and silent failure separately, because a system nobody uses can still report beautiful numbers.
Three layers, measured separately
- System health. Did it run, on every event, without gaps? Records processed against records expected, error rate, escalation rate.
- Adoption. Did a human open, read or act on the output? Callback list worked, coaching note acknowledged, flagged estimate touched.
- Outcome. Did the business result move? Booking rate, revenue per lead, unsold estimate recovery, time to respond.
Write down the baseline before you build
The most common measurement failure is that nobody recorded what things looked like before. Six months later, everyone has an opinion about whether it helped and no evidence. Baselines are cheap to capture and impossible to reconstruct after the fact.
Capture them at the same grain you will report at: by location, by channel, by team, by month. And write down the definition alongside the number, because the argument three quarters from now will be about the definition, not the arithmetic.
Where a true baseline is impossible because nobody was tracking the behavior at all, say so plainly and start measuring from launch. An honest gap is more defensible than a reconstructed number that nobody can source.
Attribution is as hard here as it is in marketing
Service businesses are seasonal and noisy. If bookings rise the same quarter you launch a call system, weather, pricing, a competitor closing and a hiring wave are all competing explanations. Claiming credit for the whole move is how AI projects lose credibility with a finance-minded owner.
Two techniques help. Stagger the rollout across locations or teams so you have a rough comparison group. And measure the intermediate step you directly control, such as the share of bookable unbooked calls that received a callback within an hour, which is far less contaminated than revenue.
Whatever method you choose, agree on it before results arrive. Measurement rules invented after a good quarter convince nobody, least of all a skeptical CFO.
Metrics that fool people
Volume of AI output is not a result. Thousands of calls analyzed proves the pipeline runs, nothing more. Model accuracy in isolation is also weak, because a highly accurate classification that nobody acts on changes nothing. Satisfaction surveys measure novelty in the first month and habit afterward.
The honest scoreboard connects a specific behavior change to a specific number in the operational system. That is why revenue intelligence is built on the job and invoice record rather than on activity counts, and why the briefs report what changed rather than how much the system processed.
Topics: measurement · baselines · roi · evaluation
Have a version of this question about your own business?
The useful answer usually depends on which systems you run and how they're connected. That's a conversation, not a blog post.