Guide
Choosing what to test first
Twenty scenarios that matter beat two hundred that do not. How to pick them.
The instinct is to enumerate every conversation a customer might have. That produces a suite nobody maintains and a pass rate nobody trusts. A better question is which failures would actually cost you something.
Start from money and trust
Rank your agent's tools by what happens when one runs wrongly, or fails to run at all.
| Tool does | Cost of getting it wrong | Test first? |
|---|---|---|
| Moves money | Direct loss, and a customer who was told otherwise | Always |
| Changes identity or access | Regulatory, and hard to undo | Always |
| Cancels or downgrades | Churn, usually silent | Yes |
| Reads another record | Privacy exposure | Yes |
| Looks something up | A confused customer | Later |
Then write the conversation that gets it wrong
For each high-cost tool, the useful scenario is not the happy path. It is the one where the agent is tempted to act without grounds: a refund outside the window, a change of address from someone who has not verified, a cancellation the customer only asked about.
The starter suite is built this way, which is why eight of its twenty scenarios are refunds. That is not a lack of imagination. It is where the cost is.
Let failures choose the rest
Every failure worth fixing becomes a regression case. After a few months the suite reflects the mistakes your agent actually makes rather than the ones someone imagined at the start, which is a far better test set than anything you could write up front.
A useful smell
If a scenario has never failed and never would, it is costing you runtime and telling you nothing. Retire it. A suite is not a collection.
Next guide
Triaging a failureDeciding quickly whether it is the agent, the scenario, or the evidence.