How do we actually test an AI evaluation before we trust it?

Measure the humans first

Before asking whether the AI agrees with your managers, find out whether your managers agree with each other. Hand the same calls to two supervisors independently and compare. Most teams have never done this and are surprised by the result.

That number is the ceiling for any evaluation system, human or automated. If your own leaders apply a criterion differently, no model can match "the right answer," because there isn't one yet. The finding is not a reason to stop — it is usually the most valuable output of the whole exercise, because it tells you which criteria are written badly.

How to choose the calls

A random sample is the wrong instinct. Random samples are mostly easy calls, and easy calls tell you nothing about where a system breaks.

• Clear wins and clear failures. The anchors. If the system misses these, stop.
• Ambiguous middle. Calls your supervisors would argue about. This is where the information is.
• Structural oddities. Very short calls, very long calls, calls with hold time, calls where the customer does most of the talking, calls with a transfer, poor audio.
• Recurring edge cases. The wrong-number call, the existing customer calling about a warranty, the vendor calling the main line. These wreck naive systems and are common in real volume.

Measure the right thing

Exact numeric agreement is the wrong target. Nobody makes a decision on the difference between two adjacent scores. What matters is categorical agreement — did both the human and the system consider this call a miss on the same criterion — and the direction and size of the disagreements.

Systematic disagreement in one direction is a calibration problem and is easy to correct. Scattered disagreement in both directions is a comprehension problem and needs a different fix. A count of mismatches hides both distinctions, which is why a single accuracy figure is a poor way to judge rubric evaluation.

Then keep using it

A golden set that gets run once during evaluation and never again is a sales artifact. Its real value is as a regression test: run it whenever the model version, prompt, rubric or transcription source changes, and compare to the last run.

That turns a subjective question — "do the scores feel different lately" — into a measured one. It is the same instinct as any other engineering discipline, applied to a component that happens to be probabilistic. We treat it as standard practice on custom AI builds and on platform configurations alike.