Testing Agentic AI: Why Output Accuracy Is Not Enough

A chatbot produces an answer. An agent changes the world.

It may search a customer record, choose a refund policy, call a billing tool, update a ticket, send a message, or hand work to another agent. The final response can look correct even when the path was wasteful, unauthorized, or dangerously dependent on luck.

That changes the test strategy. For an agentic system, output accuracy is one quality dimension—not the acceptance criterion. QA must evaluate the outcome, the sequence of decisions, tool contracts, policy compliance, resilience, latency, cost, and the system’s behavior when evidence is incomplete.

Define the agent as a controlled decision loop

Treat the system as a loop with observable boundaries:

1. 01It receives a goal and context.
2. 02It selects an action or requests more information.
3. 03A tool executes within an authorization boundary.
4. 04The environment returns an observation.
5. 05The agent continues, stops, escalates, or fails safely.

Every boundary creates a test surface. The model is only one component. Prompts, model versions, retrieval, memory, tool schemas, permission checks, orchestration, timeouts, retries, and human-approval controls can each change system behavior.

A useful test manifest therefore records the complete system version: model identifier, prompt and policy hashes, tool-schema versions, retrieval snapshot, feature flags, evaluator version, and environment seed where available. Without that manifest, a failed evaluation is difficult to reproduce and a passing result is hard to compare.

Evaluate six layers of quality

1. Outcome correctness

Did the system reach an acceptable business outcome? For deterministic tasks, compare structured fields, state transitions, totals, or database effects. For open-ended tasks, use a rubric with explicit examples and independent human calibration.

Do not reduce success to exact text matching. Two responses can differ in wording while producing the same valid result. Conversely, a polished response can accompany an incorrect or unauthorized state change.

2. Trajectory quality

A trajectory is the sequence of observations, decisions, tool calls, and results. Check whether the agent:

• —selected an appropriate tool
• —supplied valid and grounded arguments
• —avoided redundant calls and loops
• —used tool results rather than inventing facts
• —stopped when the goal was complete
• —escalated when required evidence was missing

Do not demand one golden sequence when several paths are valid. Define forbidden actions, required milestones, and acceptable ranges. A support agent may look up the order before or after reading the policy, but it must never issue a refund before verifying identity and eligibility.

3. Tool-contract correctness

Test tools like public APIs. Validate schemas at both sides, reject unknown fields, constrain enums and amounts, and make write operations idempotent. Include malformed results, partial responses, stale records, permission errors, rate limits, timeouts, and duplicate callbacks.

The orchestration layer—not the model—must enforce monetary limits, tenant boundaries, approval requirements, and destructive-action rules. A prompt instruction is not an authorization control.

4. Policy and safety invariants

Write policies as executable invariants wherever possible:

refund.amount <= policy.max_without_approval
ticket.tenant_id == authenticated_user.tenant_id
send_email requires approved_recipient == true
missing_identity_verification => no_account_mutation

Test direct violations, indirect requests, conflicting instructions, malicious tool output, and retrieved content that attempts to redirect the agent. External text is data, not trusted instruction.

5. Resilience and recovery

Inject failures between decisions and side effects. What happens when the tool succeeds but its response is lost? When the agent retries after a timeout? When a long workflow reaches its token or time budget? When a human rejects the proposed action?

Verify idempotency keys, compensating actions, resumable state, audit records, and clear escalation. A safe failure can be a higher-quality outcome than a confident guess.

6. Operational quality

Measure task completion, policy-violation rate, unnecessary tool calls, escalation quality, latency, token usage, tool cost, and human correction effort. Use percentiles and distributions rather than one average.

An agent that succeeds 96% of the time but occasionally creates a catastrophic unauthorized action is not “96% good.” Report high-severity failures separately and set a zero- or near-zero tolerance according to the risk.

Build an evaluation portfolio, not one benchmark

Use several datasets with distinct jobs:

• —Smoke set: small, deterministic, runs on every change
• —Capability set: representative normal tasks and valid variations
• —Adversarial set: prompt injection, conflicting instructions, poisoned retrieval, and abusive inputs
• —Reliability set: tool failures, latency, partial state, and retry scenarios
• —Historical set: sanitized production failures and human corrections
• —Canary set: hidden cases that protect against overfitting to the visible benchmark

Anthropic’s evaluation guidance recommends defining measurable success criteria before building tests and supports a mix of exact, programmatic, human, and model-based grading. That principle is useful regardless of model provider; see Define success criteria and build evaluations.

Protect dataset independence. Do not tune prompts against the same cases used for release approval. Track provenance, difficulty, risk category, expected invariants, and reviewer agreement.

Use model graders carefully

LLM-based graders can scale semantic review, but they introduce another probabilistic component. Give graders narrow rubrics, structured outputs, counterexamples, and an “insufficient evidence” option. Blind them to which candidate is new, randomize answer order, and calibrate against multiple human reviewers.

Use deterministic graders for schemas, tool calls, policy invariants, numeric outcomes, and state transitions. Reserve model graders for dimensions such as relevance, explanation quality, or whether an escalation is understandable.

Track grader precision and disagreement. A release gate is only as trustworthy as the evaluator behind it.

Gate releases on regressions and risk

Do not require every metric to improve. A cheaper model may slightly reduce stylistic preference while improving latency with no material risk. Define:

• —hard blockers for policy or authorization violations
• —non-inferiority thresholds for critical capabilities
• —statistical or repeated-run criteria for nondeterministic outcomes
• —cost and latency budgets
• —mandatory human review for ambiguous high-impact regressions

Run important cases multiple times and report pass-attempt distributions. Preserve traces for failures and a sample of passes. OpenTelemetry’s evolving generative-AI conventions provide common concepts for recording model operations, tool calls, results, usage, and evaluation signals; see OpenTelemetry semantic conventions.

The central QA question for an agent is not, “Did it say the right thing?” It is, “Did the entire controlled system reach an acceptable outcome through an authorized, robust, and explainable path?” Once that becomes the test oracle, agent quality stops being a demo metric and becomes an engineering discipline.