What happens to our call scores when the AI model gets updated?
They can shift, and if nobody is watching you will read a model change as a performance change. Any evaluation system needs a frozen set of previously reviewed calls that is re-run whenever the model, prompt or rubric changes, so you can compare new results against old ones on identical inputs. Stamp every stored result with the model and rubric version, or your history stops being interpretable.
The scenario worth preventing
A manager notices that team scores dropped across the board last month and starts a round of difficult conversations. Nothing about the team changed. A provider shipped a model update, or someone edited the prompt, and the scoring got slightly stricter on one criterion.
This is not exotic. It is the default outcome of running evaluation without version control, and the damage lands on people who did nothing wrong.
Build a regression set and freeze it
Take a set of calls — a few dozen to a couple hundred, depending on how many criteria you score — and have humans review them carefully. Keep the range wide: strong calls, weak calls, noisy audio, transfers, short calls, the ambiguous ones people argued about.
Freeze it. When anything changes, re-run the whole set and compare against the human labels and against the previous run. You are looking for two things: overall agreement, and per-criterion movement. A change that improves the average while breaking one criterion is easy to miss without the breakdown. This is the same discipline that keeps operational AI trustworthy over time.
Stamp every record with its versions
Every stored result should carry the model identifier, the prompt version and the rubric version that produced it. Without those, a year-over-year comparison is comparing measurements taken with different instruments.
Version the rubric like code, with a date and a note on what changed and why. Rubrics get edited constantly in the first months of a program, usually by a manager fixing a wording problem in a hurry, and those edits move scores just as surely as a model change does.
With them, you can filter reporting to a consistent version window, explain a step change in a chart, and answer the only question that matters when someone challenges a historical score: what exactly produced this?
Pinning and upgrading both have costs
Pinning to a fixed model version buys stability and eventually strands you on something deprecated. Auto-upgrading buys improvements and moves your baseline without warning. Neither is free.
The workable pattern is to pin, test upgrades against the regression set on a schedule, and adopt deliberately — with a note in the reporting marking the date the version changed. If your rubric or prompts live inside a configurable platform rather than scattered through code, that cycle is a routine task instead of a project.
Topics: model versions · drift · regression testing · evaluation
Have a version of this question about your own business?
The useful answer usually depends on which systems you run and how they're connected. That's a conversation, not a blog post.