How do I audit what an AI system did last month?
You need three linked records for every action: the input it saw, the output it produced, and the effect it had on another system, each timestamped and stamped with the model and prompt version in use. Miss any one and you can describe the system's general behavior but cannot reconstruct a specific decision, which is exactly what you need when somebody disputes one.
The three records, and why all three
The input record proves what the system knew. Without it, a wrong output looks like a model failure when it may have been a data failure upstream. The output record proves what it concluded. The effect record proves what changed as a result, which is the only part that shows up in the business.
Systems commonly log the middle one and neither of the others. That is enough to say "the AI classified this call as a sales opportunity" and not enough to say what it heard or what happened next.
Version stamps are the part everyone skips
Six months later, a disputed result is meaningless without knowing which model version, which prompt revision and which rule set produced it. Prompts change more often than anyone expects, and a change made on a Tuesday afternoon can silently invalidate a comparison across the whole quarter.
Treat the prompt and the ruleset as versioned artifacts, the same way you treat code. Stamp the version onto every output. This is cheap to add and impossible to reconstruct after the fact.
A useful shortcut is to store the fully resolved prompt exactly as it was sent, not the template it came from. Templates get edited, and a template plus a set of variables you no longer have does not reconstruct what the model actually saw on the day in question.
What audits are usually actually for
- A disputed customer interaction. Someone says the system did something; you need the record, not a recollection.
- A reporting discrepancy. Two numbers disagree, and you need to trace one back to the events behind it.
- A manager questioning a score. Common with evaluation work, and entirely reasonable. The answer should be the evidence the score was based on.
- A periodic quality review. Sampling old decisions to see whether quality has drifted since the last model change.
Retention is a decision, not a default
How long you keep transcripts, recordings and model inputs should be chosen deliberately, because holding them has obligations attached and discarding them removes your ability to audit. Requirements for recording, consent and retention vary by state and by industry, and they change; treat this as a question for your own counsel rather than something to settle from a vendor's documentation.
The engineering side is straightforward once the policy exists. It is the policy that has to come first, and it should be written down next to the rest of your call intelligence and operational AI configuration.
Topics: audit · logging · traceability · governance
Have a version of this question about your own business?
The useful answer usually depends on which systems you run and how they're connected. That's a conversation, not a blog post.