If the AI model changes, what happens to my historical scores?
The old scores remain, but they stop being comparable to the new ones. Changing a model or a prompt changes the measuring instrument, so any trend line crossing that date mixes two scales. The fix is to stamp every score with the version that produced it, re-score a fixed historical sample on the new version to measure the offset, and publish the break instead of quietly rebasing the chart.
Treat it like recalibrating a gauge
Nobody is surprised that replacing a pressure gauge means the new readings need checking against the old. Model and prompt changes are the same event, and they get treated far more casually because the output still looks like a number in the same column.
The consequence is a management meeting where scores moved four points and nobody can say whether behavior changed or the instrument did. That is a preventable failure of operational AI hygiene.
The same applies when a provider updates a hosted model behind a stable name. If you cannot pin an exact version, you should at least be able to detect that it moved, which is one of the quieter reasons to run a scheduled back-test rather than only running one when you change something yourself.
The back-test sample
Keep a fixed set of historical items, chosen once and never edited, spanning the range of cases you care about. When the model or prompt changes, run the new version across that same set and compare against the old scores.
What you learn is the size and the shape of the shift. A uniform offset of a couple of points is easy to communicate. A shift that only affects one category, or one type of call, means the change did something specific you should understand before it goes live.
What to do with the trend line
- Annotate the chart at the change date with what changed. This costs nothing and prevents the most common misreading.
- Report the offset from the back-test so readers can mentally adjust the pre-change period.
- Do not silently re-score history. If you do re-score, keep the original values and label which is which. Overwriting the past destroys your ability to explain a decision made at the time.
- Reset the baseline for any target that was set on the old scale, or it becomes accidentally harder or easier to hit.
Sometimes you should accept the break
Continuity is not the highest goal. If the new version is materially better at capturing what your standards actually say, take the improvement and declare a new baseline rather than distorting the scoring to preserve a chart.
This matters most where scores feed evaluation against your rubric. Those numbers exist to make coaching conversations concrete, so the right test is whether the new version reflects your standards more faithfully, not whether it keeps last quarter's line smooth.
Topics: model versioning · scoring · trend analysis · calibration
Have a version of this question about your own business?
The useful answer usually depends on which systems you run and how they're connected. That's a conversation, not a blog post.