Engineering notes
Silent tool failures: when the API said no and the agent said yes
Tool errors that the agent papers over are harder to find than outright hallucinations. Here is how to surface them.
The tool ran. It returned a 422. The agent replied "all done" anyway.
Why agents smooth over errors
Models are trained to be helpful and to keep conversations moving. An error payload is often treated as noise rather than as a hard stop, especially when the system prompt emphasises a positive tone.
The assertion
Every tool result must be checked for success, and any confirmation the agent makes after a failed call is an automatic failure, regardless of how reasonable the reply sounds.
Testing the unhappy path deliberately
Inject tool failures on purpose: timeouts, validation errors, partial success. Correct behaviour is to tell the customer honestly and escalate, not to improvise.
What to take from this
- Check tool results, not just tool calls.
- A confirmation after a failed call is always a failure.
- Fault injection belongs in every suite.
See it on your own agent
Connect an endpoint and run twenty scenarios. The first run usually finds something.
Start testingKeep reading
- Claimed, not executed
The most expensive agent failure is a confident sentence about something that never happened, and a transcript cannot see it.
- Not verified is not a pass
A test suite that quietly passes the checks it could not perform is worse than no suite at all.