Two managers scored the same call differently. Who is right?

Compare items, never totals

Two managers can land within a few points of each other on the total score while disagreeing on half the individual items, with the errors cancelling out. The total looks reassuring and hides the problem completely.

Calibration only works at the item level. Line up both scorecards side by side and count how many items matched. That number, not the score gap, tells you whether your rubric is functioning.

Track the agreement number over time and treat it as a health metric for the rubric itself. If two managers agree on nine of twelve items, the three they split on are your entire backlog.

How to run a calibration session

• Pick five calls that span the range: one clean booking, one clear miss, one angry caller, one price shopper, one messy edge case.
• Score independently, without discussion, and without seeing each other's marks.
• Compare item by item and stop on every disagreement.
• Rewrite the item, not the person's opinion. If two reasonable people read it differently, the sentence is at fault.
• Re-run on new calls a month later to confirm the rewrite worked.

Items that stay contested after two rewrites

Some items resist definition because the underlying idea is genuinely subjective. Showed empathy is the classic case. You have three options: replace it with an observable proxy, demote it to a note that carries no score, or drop it.

Dropping an item feels like lowering standards. It is not. An item that cannot be scored consistently was never enforcing a standard, only creating noise.

Be honest about the cost of keeping a contested item. Every ambiguous line on the scorecard adds noise to every rep's total, which makes real differences between people harder to see.

Where automated scoring helps, and where it does not

A model applying the same rubric to every call is consistent by construction. That is not the same as being correct, and the distinction matters: consistency means the comparison between two reps is fair, but the standard itself can still be wrong.

So the calibration work does not disappear when you automate — it moves. Instead of calibrating managers against each other, you periodically sample scored calls and check them against a human read, then adjust the rubric wording where the two diverge. That audit loop is a permanent part of running AI rubric analysis responsibly, and it belongs to the same coaching process everything else does.