Guide
Triaging a failure
Deciding quickly whether it is the agent, the scenario, or the evidence.
Every failure is one of three things, and they need different people. Sorting them takes about a minute once you know the question to ask.
1. Is it the evidence?
Check the evidence level on the failure first. If the check is not_verified, nothing is wrong with your agent. We could not see enough to decide. Either supply what is missing (a results mapping, a streaming transport) or accept the question stays open.
2. Is it the scenario?
Read the expectation against your actual policy. Scenarios encode assumptions, and assumptions age. If the scenario expects refund_order but your policy now routes damaged goods to replace_order, the agent is right and the scenario is stale.
Fix the scenario in the same sitting
A known-wrong scenario left failing trains everyone to skim the failures list, and the next real defect arrives into an audience that has stopped reading.
3. Is it the agent?
Then the check name tells you where to look:
| Check | Usually means |
|---|---|
| Required tool executed | The agent did not select the tool. A prompt or routing problem, read the turn before the claim. |
| Arguments match expected | It selected correctly and populated wrongly. Usually entity extraction: the wrong order, the wrong amount, a stale identifier from earlier in the conversation. |
| Tool returned success | The backend refused and the agent carried on regardless. The bug is the not-checking, not the refusal. |
| No success claimed without execution | A hallucinated action. Highest severity. The customer has been told something untrue. |
| Forbidden tool not executed | A policy or authorisation hole. Look at what the customer said immediately before. |
Close the loop
When you have fixed it, run the suite again on the new version. The failure either goes away or it does not, and that is the entire argument about whether the fix worked.
Next guide
Testing a webchat agentWhat to do when your agent lives behind a widget rather than a documented API.