Our AI scores shifted and nothing in the business changed. What happened?
Something in the system moved: the model version rolled forward, a prompt was edited, the rubric wording changed, or the transcription engine was upgraded. The way to prove it is to re-score a frozen set of old calls with the current configuration. If scores move on inputs that did not change, the shift is yours, not the market's. This is why every output should be stamped with model, prompt and rubric version.
Four candidate causes, in order of likelihood
- Model version changed. Providers deprecate and replace models. A vendor that does not pin versions inherits behavior changes silently.
- Prompt or rubric edited. Someone reworded a criterion to be clearer. Clearer often means stricter. Nobody logs it as a change to a metric definition, but that is what it was.
- Input changed. A new transcription engine, a new phone vendor, a change in how much of the call is captured. Better transcription can lower scores by exposing content the old one missed.
- The business actually changed. Possible, and worth ruling out last, because it is the only one you cannot fix in software.
The frozen-set test
Keep a fixed set of past calls — enough to be more than anecdote, spanning good, bad and ambiguous. Whenever the configuration changes, re-score that set and compare to the stored prior scores.
Movement on the frozen set is definitive. Those calls did not change. If their scores did, the system did. This single practice resolves most arguments about whether a metric shift is real, and it takes an afternoon to set up.
Version stamping is cheap and non-negotiable
Every stored evaluation should carry the model identifier, the prompt version and the rubric version that produced it. Without those three fields, a trend chart spanning a configuration change is comparing two different instruments and presenting the difference as a business result.
Once you have them, the fix for a confusing trend is easy: split the chart at the version boundary, or annotate it. An annotated discontinuity is honest. An unannotated one has caused more bad management decisions than any model error.
How to handle a version change you want
Upgrades are usually worth taking — newer models generally read transcripts better. The discipline is to take them deliberately. Re-score the frozen set, quantify the shift, decide whether to re-baseline history or start a new series, and tell the people whose numbers are about to move before they move.
Reps notice score shifts immediately and interpret them as being graded harder for no reason. Announcing a model change and explaining the re-baseline preserves the credibility that coaching programs depend on. Version discipline is part of what we mean by operational AI: the system is instrumentation, and instrumentation has to be calibrated and dated.
Topics: model versioning · drift · diagnostics · reporting
Have a version of this question about your own business?
The useful answer usually depends on which systems you run and how they're connected. That's a conversation, not a blog post.