How do I audit whether our AI receptionist is actually doing a good job?
Do not listen to random calls. Random sampling spends your attention on the calls that went fine. Build a stratified sample from the calls most likely to contain failure: transfers, very short calls, very long calls, calls where the agent expressed uncertainty, and calls followed by a repeat contact within two days. Score them against a written rubric, and re-sample the same strata every month so you can see drift.
Random sampling wastes the reviewer
If most calls go acceptably, a random sample of twenty calls mostly confirms that most calls go acceptably. You learn almost nothing, and the exercise dies after two months because it feels pointless. The failures live in a small, identifiable subset, and that subset can be selected for.
Stratified sampling means deliberately over-weighting the risky strata, reviewing them, and being honest that the resulting scores are not a population estimate. You are hunting failure modes, not producing a grade.
The five strata worth pulling
- Transfers. Especially transfers that did not complete. Every one is a case the agent could not handle plus a case a human may not have caught.
- Calls under about thirty seconds. Short calls are either wrong numbers or abandonments, and the ratio between them matters.
- Calls in the longest decile. Long calls usually mean loops, repeated confirmations or a caller fighting the system.
- Calls containing uncertainty language. Any turn where the agent said it did not understand, or asked the same question twice.
- Calls followed by a repeat contact within forty-eight hours. The single highest-yield stratum, because it identifies calls that appeared successful and were not.
Score against something written down
Reviewer impressions do not aggregate. Two managers listening to the same call will disagree about whether the agent was "helpful". A rubric with concrete, observable criteria produces scores that can be compared across months and across reviewers, which is the entire point of rubric-based evaluation: your standards, applied consistently.
Keep the rubric short. Six to eight criteria that each have a clear yes or no. Did it capture the address correctly. Did it identify the actual reason for the call. Did it escalate when it should have. Did it make a commitment the business can keep.
Watch for drift, not just level
The useful output of an audit is not this month's score. It is the comparison to last month's score on the same strata. Voice agents drift for mundane reasons: a prompt was edited, a model version changed, a hold list expired, a business rule moved. None of those announce themselves.
Pair the monthly human audit with an automated check that runs on every call, so the audit is confirming a signal rather than discovering one. That combination, machine review of everything plus human review of the suspicious slice, is how call intelligence is meant to be used, and it is the same pattern that works for coaching human reps.
Topics: QA · auditing · sampling · rubric · voice AI
Have a version of this question about your own business?
The useful answer usually depends on which systems you run and how they're connected. That's a conversation, not a blog post.