How Do You Test an Agent That Never Answers the Same Way Twice?

The first thing that breaks when a team starts shipping agents is the test suite. Not because the agents are unreliable, but because equality assertions stop meaning anything.

assertEquals(expected, actual) assumes a function with one right answer. An agent asked the same question twice will phrase it differently both times and may be entirely correct both times. The test fails, nothing is wrong, and within a fortnight somebody deletes the test.

The failure mode after that is worse: the team concludes agents cannot be tested and starts shipping on vibes.

They can be tested. The assertions have to move from the wording to the properties.

Assert on properties, not strings

For any given output there are usually a handful of things that must be true regardless of phrasing. Those are the assertions.

Did the answer cite a source? Does the cited source exist? Does it contain the claim attributed to it? Did the agent stay within scope? Did it decline to state a figure it does not hold? Did it call the tool it needed, with arguments derived from the conversation rather than invented?

Each of those is checkable mechanically and none depends on how the sentence was constructed. A suite built this way is stable across model versions and prompt edits, which is the property that makes it worth maintaining.

Test the guardrails as code, not as prose

A guardrail written into a system prompt is a request. It is a good request, models generally honour it, and it is not a control — because a prompt can be argued with and because a prompt can silently stop being included at all.

That last one is not hypothetical. We shipped a voice agent whose system prompt template never interpolated the guardrail block. Nothing failed and nothing warned. The rules against quoting rates, stating exact headcounts and naming clients under NDA applied to the typed chat and not to the voice — which was the primary product. It survived several passes over both files, because two prompts that read similarly are hard to diff by eye.

The fix was not more review. It was a build step that asserts each rule is present in the text and wired into both prompts, and fails the build otherwise. Nothing about a missing prompt fragment is detectable at runtime, and by the time it is detectable it has been said out loud to somebody.

If a rule matters, something other than a person's attention has to check it is still there.

Adversarial cases belong in the suite

Every public agent will be probed. Someone will try "ignore your previous instructions", someone will claim to be an administrator, and someone will paste a document containing text aimed at the reader rather than about the subject.

These are test cases. Write them down, run them on every build, and assert on behaviour: the agent stays in role, does not disclose its instructions, does not follow instructions found in retrieved content, and — importantly — does not become hostile or start lecturing, because an agent that accuses a curious developer of attacking it is its own kind of failure.

Treating retrieved text as data rather than instruction is worth its own test. If a document says to disregard the rules, that is a fact about the document.

Filter where filtering is still possible

Test coverage has to follow the architecture, and the architecture decides where a control can exist at all.

A typed reply is composed on your server, so it can be screened on the way out. Spoken audio streaming from a model provider straight to a browser cannot — there is no moment afterwards when anything can intervene. For that path the last controllable point is what you hand the model, which means tool results get filtered before the model sees them, and that filter needs tests of its own.

Knowing where the last controllable point is, per path, is most of designing the test plan.

Regression by sampling

Some qualities genuinely resist assertion: tone, helpfulness, whether an answer is useful rather than merely correct. Sample these rather than abandoning them. A fixed set of representative conversations, run each release, reviewed by a person against the previous release's output.

It is not automated and it does not need to be. Twenty conversations reviewed properly catch more real regressions than a thousand assertions on string equality ever will.

---

iLeaf builds and operates agentic systems in production, including AgentOps and MLOps for teams running agents they need to keep trusting.