What twenty hand-written scripts missed

We validate the detector against a reference agent with deliberately planted faults. If the suite finds the planted bugs and nothing else, the detector works. Twenty hand-written scenarios, three known faults, three failures. Clean.

Then we swapped the scripted customer for a model playing the same personas, same scenarios, same expectations, same agent, and ran it three times. Sixty conversations, not one byte-identical to another.

The result we wanted

Zero verdict flips. Every scenario reached the same pass or fail conclusion across all three runs, despite the conversations differing every time. That matters: a non-deterministic driver is only usable if the verdicts are stable, otherwise you have built a coin toss with a progress bar.

The result we did not expect

It found three more defects. Real ones, in code we wrote and believed we understood:

• A privacy leak - asked about a different customer's order mid-conversation, the agent called check_shipping on it. No script had ever changed subject that way.
• An identity failure - during a data-fishing attempt, the agent ran update_address for an unverified caller.
• A premature action - a customer who opened with *Okay, but before anything…* had their refund processed while still asking a question. The scripts always confirmed cleanly, so the confirmation check had never seen a hedge.

Each of these is a hole in the *scripts*, not in the evaluator. We wrote scenarios covering the failures we could imagine. A customer who wanders is an excellent generator of the ones we could not.

The honest trade-off

Generated customers cost model tokens, take longer, and produce transcripts you have not read before. Scripts are free, instant and identical. Neither is the right default for everything.

What we run: scripts as the release gate, because a blocking check should never be the thing that flakes. Generated customers on a schedule, because that is where the next unknown failure is. The ground-truth files in our own repo now carry a note recording which driver found which bug, otherwise the next person to read them draws the wrong conclusion about coverage.