For decades, QA has run on one quiet promise: same input, same code, same output. That’s what made test suites worth writing. It’s why regression testing worked, and why a green build meant something.
Agentic AI breaks that promise.
Picture an agent that passes 1,000 security evaluations in staging, then makes an unauthorized transaction on run 1,001. Nobody had to change the code. Maybe the prompt was phrased a little differently, or a retrieved document was formatted oddly, or a tool returned something unexpected, and the agent drifted onto a different line of reasoning. And it didn’t just give a bad answer. It did something in the real world.
That’s the core problem. We’re trying to assure systems whose behavior is probabilistic, context-dependent, multi-step, and consequential, using a discipline built for determinism.
The old QA contract
Traditional testing ran on a simple deal. Define an input, define the expected output, assert it. A regression suite tells you when a change breaks something that used to work. Staging approximates production well enough to catch most problems before release. And if transfer_funds(amount=10000) gets blocked for an unauthorized role in staging, you can reasonably expect the same rule to hold in production, as long as nothing in the code, config, or environment changed.
None of this meant software was perfectly predictable. Distributed systems fail, race conditions happen, and production data always finds a way to surprise you. But QA could shrink the uncertainty by isolating variables and controlling inputs. It didn’t eliminate uncertainty so much as shape it into something manageable and repeatable.
What changes with agents
An LLM agent isn’t a function that turns an input into an answer. It plans, retrieves knowledge, calls APIs, writes to memory, delegates, and acts across connected systems. So its behavior isn’t just an output. It’s a trajectory: the chain of reasoning steps, tool calls, retrieved context, and intermediate results that led to the final response. That’s why evaluation transcripts should capture all of it, not only the last message.
A few things make this much harder to test:
- Stochastic inference. Similar runs can produce different words, plans, or tool calls.
- Prompt drift. Small changes to system instructions, user phrasing, policy text, or tool descriptions can shift behavior.
- Contextual hallucination. The agent invents a plausible premise and then proceeds as if it were fact.
- Tool-use variance. Given the same task, it may pick different tools, arguments, or orderings.
- Feedback loops. A database row, web page, or API response can redirect its next decision.
- Compounding. A small misreading early on can turn into an unauthorized action several steps later.
So “the agent passed security testing” is really an incomplete sentence. It passed a sample of testing under particular conditions. NIST’s AI risk guidance says as much: test before deployment and regularly during operation, and show performance under conditions that resemble the real deployment.
An agent can reason incorrectly, act incorrectly, and leave consequences behind.
The 1,001st run
Take a finance operations agent. The workflow is simple. An employee asks it to prepare a vendor payment. The agent pulls the invoice, checks the vendor details, and drafts a payment instruction. If the amount is over the policy threshold, it must route to a human for approval. Only then can the payments system execute.
In staging, the team runs 1,000 scenarios. The agent spots the approval requirements, refuses excessive amounts, cites the policy, and only calls the payment tool once it has an approval token.
Then comes run 1,001. The user’s wording is a bit different. The retrieved policy excerpt is formatted differently. A vendor record has an ambiguous field. The agent reads “please process this urgently” as implicit authorization, calls the payments API with plausible arguments, and the execution layer says yes. The money moves.
No line of code changed, and there’s no obvious regression to point at. Yet the system crossed a line it was supposed to guard.
It probably wasn’t one dramatic hallucination, either. More likely it was a pile of ordinary uncertainties: ambiguous wording, a misleading document, a loosely scoped tool, a stale memory entry, and an execution layer that trusted the model’s interpretation instead of enforcing policy on its own.
Staging success isn’t proof of production safety. It’s a sample from a distribution of possible behaviors.
Why pass/fail breaks
The deeper issue is conceptual. Traditional QA asks whether a system passes. With agents, the harder question is: under what distribution of conditions, across how many trials, and along which execution paths does it stay safe?
Put the old assumptions next to the new reality and they barely line up. One test run used to be enough; now it’s one sample from a behavioral distribution. A pass used to mean reliable behavior; now it means the agent succeeded this time. Failures used to trace back to a code change; now they can come from prompts, context, retrieval, tools, memory, or the environment. Coverage used to mean code paths; now you need coverage of trajectories, tool calls, memory states, and side effects. Security checks used to be static; now authorization has to be enforced at execution time, outside the model’s discretion. And a green build used to mean release-ready; now it means the tested slice behaved acceptably under the tested conditions.
The most dangerous mistake is treating an agent evaluation like a unit test. A unit test says “this function behaves as specified.” An agent evaluation says “in this trial, with this context and these tools, the agent took an acceptable path.” Those are different claims.
The real engineering shift
The answer isn’t to give up on QA. It’s to move from deterministic verification toward probabilistic assurance.
Test trajectories, not just answers
A correct final answer can hide an unsafe path. An agent might land on the right invoice total while leaking sensitive data, calling a tool it shouldn’t, skipping an approval step, or using more privilege than it needed. So look at the whole path: tool selection, arguments, retrieved sources, policy checkpoints, memory operations, handoffs, error recovery, side effects.
The useful question isn’t only “did it get the right answer?” It’s “did it get there in a way we’d defend in an audit, an incident review, or a regulator’s inquiry?”
Measure consistency, not occasional success
Repeating trials turns evaluation from a yes/no event into a reliability measurement. Two metrics help here:
- Pass@k is the chance that at least one of k attempts succeeds. It measures capability: can the agent ever solve this?
- Pass^k is the chance that all k attempts succeed. It measures consistency: can it solve this every time?
Anthropic notes that pass@k climbs as you allow more attempts, while pass^k drops as you demand more consistency. An agent with a 75% per-trial success rate has only about a 42% chance of passing three in a row (0.75³ ≈ 0.42).
For a coding assistant, pass@k can be fine, since one good solution may be enough. For a payments, support, or sysadmin agent, pass^k is much closer to what users and regulators actually expect. Occasional competence isn’t operational reliability.
Put deterministic guards around probabilistic behavior
The model may be non-deterministic. The system around it shouldn’t be. Don’t count on the model to remember its permissions, interpret policy correctly, or politely decline something dangerous. Put hard controls outside its context:
- Validate every tool call against a schema.
- Enforce authorization in the execution component, not just in the prompt.
- Use allowlists for tools, recipients, resources, environments, and transaction types.
- Scope tools by least privilege, keeping read-only separate from write and admin access.
- Require explicit, action-specific approval for anything irreversible or high-impact.
- Separate deciding from executing: the agent proposes, and a policy service independently checks scope, privilege, and approval state.
- Bind approvals to the exact actor, tool, target, parameters, timestamp, and expiry, and consume them atomically before execution.
A user_confirmed flag sitting in the agent’s context isn’t enough. The execution component has to verify that the approval belongs to this actor and this exact action, is still valid, and hasn’t been used already.
The model can propose. The system has to decide.
A practical testing stack
You don’t need one “agent QA” gate. You need layers.
1. Deterministic assertions. Wherever the expected behavior can be stated precisely, check it in code: schema validation on tool arguments, authorization checks before execution, spend limits and allowlists, database-state assertions after a workflow, hard constraints on destructive operations, and audit records for every consequential action. None of this should depend on the model’s judgment.
2. Trajectory evaluations. Inspect the full path: which tools were chosen, what arguments went in, what was retrieved, whether policy checkpoints held, what memory was read or written, whether the agent escalated when it should have, how it recovered from errors, and what it changed in external systems. This matters most for security. A malicious instruction hidden in a retrieved document, memory entry, or tool response can redirect an agent later, and memory and context poisoning are recognized agentic risks.
3. Distributional testing. Run high-risk scenarios over and over, varying user phrasing, conversation length, retrieved documents, tool responses, error conditions, memory state, task order, and adversarial injections. Report success rates, variance, failure categories, and how often unsafe paths show up. NIST’s playbook suggests documenting variance with things like confidence intervals, standard deviation, bootstrapping, or cross-validation. A single green run isn’t an evaluation. It’s one observation.
4. Adversarial and security testing. Go beyond “can the agent refuse a malicious request?” and test whether the surrounding system holds up under manipulation: tool misuse, privilege escalation, approval bypass, memory poisoning, injection through retrieved content, data exfiltration, credential reuse across tasks, cross-agent impersonation, and tool sequences that each look permitted but are harmful together. OWASP’s guidance leans on least privilege, per-tool permission scoping, separate toolsets by trust level, and authorization enforced outside the agent’s context.
5. Production observability. Staging can’t reproduce everything production throws at you. Capture prompts, retrieved context, tool calls and arguments, policy decisions, outputs, human approvals, downstream effects, and user feedback. NIST calls for monitoring AI behavior in production, not just testing beforehand. Anthropic makes a similar point: evals make behavior changes visible before users feel them, but monitoring is still essential for what static benchmarks miss.
What “good enough” means
For conventional software, good enough usually means the suite passes and known regressions are gone. For agents it has to mean more:
- The agent meets a defined reliability threshold across repeated trials.
- Deterministic controls block unsafe trajectories.
- High-impact actions need explicit authorization.
- Failures degrade safely instead of executing something irreversible.
- Teams can reconstruct what the agent did, why, and which control failed.
- The system can be contained, overridden, or shut off when behavior diverges from intent.
The goal isn’t a perfectly predictable agent, which may be impossible. It’s making the unacceptable behaviors impossible, observable, reversible, or contained.
QA is not dead, its assumptions are
There are two tempting reactions to all this. One says agents can’t be reliably tested, so don’t deploy them. The other says it passed evaluation, so it’s safe. Both are wrong.
Agents can be evaluated, but it takes a different model of assurance: distributions instead of single runs, trajectories instead of final answers, execution-time enforcement instead of trust in the prompt, and continuous monitoring instead of a one-time release gate.
The question is no longer “did the agent pass the test?” It’s “can we trust the system when the agent takes a path we didn’t anticipate?”
That’s the real QA challenge of the agentic era.