How do we know if the AI receptionist is actually working?
Ignore containment rate. The share of calls the agent handled without a human is a measure of how often it refused to transfer, not of whether it did the job. Measure booked jobs per hundred bookable calls, the correction rate on what it wrote, mid-conversation hang-ups, and repeat contacts within a day or two — and compare all of it against the same call population handled the old way.
Why containment misleads
Containment is popular because it is easy to compute and it always looks good. But an agent maximizes containment by never transferring — including on the calls that should have been transferred immediately.
You can raise containment by removing the human exit, and every one of those retained calls that ends in a frustrated hang-up counts as a success. Any metric that improves when the system behaves worse is not a metric, it is a scoreboard.
The four numbers that matter
- Booked jobs per hundred bookable calls. Bookable is the key word — exclude wrong numbers, vendors, and existing-job status checks, or the denominator hides everything.
- Downstream correction rate. How often a human edited the job the agent created, and which field. This is your true accuracy signal.
- Mid-conversation abandonment. Callers who hang up while talking to the agent. This is the frustration measure containment hides.
- Repeat contact within two days. Someone calling back about the same thing means the first interaction did not resolve it, whatever the transcript says.
Pick the right comparison
The most common analysis error is comparing agent-handled calls to your team's overall performance. The agent usually gets nights, weekends, storm-day overflow and second-choice callers. Your team gets Tuesday morning.
Compare like to like: agent-handled after-hours calls against your previous after-hours handling, agent overflow against voicemail return rates. If you want a genuine head-to-head, split the same call type and the same hours between agent and human for a period. Everything else is a story, not a measurement. This is the same care that performance comparison across reps requires.
Seasonality will fool you as well. A month-over-month improvement during the start of peak season is mostly weather, and a decline in a slow month is mostly demand. Hold the comparison window and the call mix constant or the number means nothing.
Read the calls, not just the dashboard
Numbers tell you something changed. Transcripts tell you why. A weekly habit of listening to a handful of abandoned calls and a handful of corrected bookings surfaces problems that no aggregate would show — a phrase the agent mishandles, a question it keeps dodging, a job type it consistently misclassifies.
Running that review systematically across all calls rather than a sample is exactly what call intelligence is for, and it applies to human-handled calls just as much.
Topics: metrics · containment · measurement · benchmarks
Have a version of this question about your own business?
The useful answer usually depends on which systems you run and how they're connected. That's a conversation, not a blog post.