What a Test Plan for an AI Feature Actually Contains

The first honest problem with testing an AI feature is that the usual method does not apply. You cannot call it twice and assert the outputs match, because they will not, and that is by design rather than a defect.

A lot of teams stop there and fall back to a person trying it before each release. That is not a test plan, it is a habit, and it catches only the failures obvious enough to notice in ten minutes.

Here is what actually goes in the plan.

Separate what is deterministic from what is not

Most of an AI feature is ordinary software and should be tested as such. What gets retrieved and in what order. How context is assembled and truncated when it exceeds the limit. How the response is parsed. What happens when the response is empty, malformed, or valid JSON with the wrong shape. What the tool-calling loop does when the model requests a tool that does not exist, or the same tool four times in a row.

All of that is deterministic if you mock the model, and all of it is where a surprising share of production incidents come from. Test it conventionally, with conventional assertions, at conventional coverage.

What remains genuinely non-deterministic is the model's output, and that is a smaller surface than it first appears.

Build a graded set, not a smoke test

For the model itself you need a fixed collection of inputs with known correct behaviour, and it has to be built deliberately rather than sampled casually.

Three groups matter. Ordinary cases, which the feature is expected to get right, drawn from real usage rather than invented. Hard cases — ambiguous, long, unusual, contradictory input — which are where accuracy actually degrades and where the business cost of an error is usually highest. And out-of-scope cases, which the system is expected to decline, because refusal behaviour is a feature and it regresses silently.

Fifty well-chosen cases beat five hundred sampled at random. The value is in the hard and out-of-scope groups, and those have to be curated by someone who understands the domain.

Assert on properties, not on text

Since you cannot compare to a reference string, assert the things that must be true of any acceptable answer.

Does the output parse. Does it carry a citation when the answer requires one, and does that citation point to a document that exists and actually contains the claim. Does it stay within the permitted set of actions. Does it decline when the retrieved context does not support an answer. Does it avoid asserting anything about entities not present in the input.

That last check catches the failure that matters most. A system inventing a plausible detail is far more dangerous than one producing an awkward sentence, and it is detectable: compare the entities and figures in the output against those in the retrieved context.

Track a rate, not a pass

These tests are probabilistic, so a single failure is not a regression and a single pass is not a guarantee. Run the set on every meaningful change, record the pass rate per group, and watch the trend.

Set the gate on the rate — ordinary cases above some threshold, out-of-scope refusals essentially absolute — and treat a drop as a release blocker in the same way a failed integration test would be. The important discipline is that the number is recorded every time, because the characteristic failure of AI features is slow drift, and drift is invisible unless something has been counting.

Test the seams around the model

Two areas are consistently under-tested and consistently break.

Behaviour under provider failure. Time out the model call, return a 500, return a rate-limit response, return a truncated body. The feature should degrade to something defined. A great many AI features have no tested behaviour here at all, and discover their actual behaviour in an incident.

Prompt injection through content. If the system ingests documents, emails or user-supplied text, someone will eventually put instructions in that content. Put them in yourself first: a document containing text that tells the assistant to ignore its rules, or to reveal its configuration, or to call a tool it should not. The system should treat retrieved content as data. Verifying that is a test, and it belongs in the suite rather than in a security review a year later.

What the release actually gates on

In practice the plan comes down to four things: the deterministic surround is covered properly, the graded set runs automatically with its rate recorded, refusal and citation behaviour are held to a hard threshold, and provider failure has a defined outcome.

That is achievable, it is maintainable, and it catches the failures that reach users. It is also considerably more than a person trying it before the release, which remains the industry norm.