Guide
Tool validation patterns
Assertions for call presence, ordering, arguments, idempotency and failure handling, with the trade-offs of each.
Tool assertions are the highest-value checks you can write, and the easiest to write badly.
Presence
The required call exists in the trace. Keep this loose about ordering unless ordering genuinely matters, over-specified tests break on harmless refactors.
Ordering
Assert order only where it is semantically required: verify identity before disclosing account data, check policy before issuing a refund.
Arguments
Validate against conversation context, not against fixed literals. The amount should match the order the customer discussed, whichever order the scenario generated.
Idempotency
Assert the money-moving call happened exactly once. Duplicate refunds are a real and expensive agent failure.
Failure handling
Inject an error and assert the agent tells the truth and escalates rather than confirming success.
What to take from this
- Assert presence broadly, ordering narrowly.
- Validate arguments against context, not constants.
- Exactly-once matters for anything touching money.
Next guide
Rolling out agent testing across a teamWho owns scenarios, who triages failures and how to introduce gating without stalling every release in week one.