Pass rate is the wrong number

Every agent testing tool leads with a pass rate, and it is close to useless as a release signal.

Consider two candidate versions, each scoring 85% against a twenty-scenario suite. The first fixed three failures and broke none. The second fixed three and broke three. Same number. One is a good release and the other should not ship.

What a release decision actually needs

Not a score but a classification, scenario by scenario, against the version you are replacing:

• Fixed - failed before, passes now. The reason you are shipping.
• Regressed - passed before, fails now. The reason not to.
• Still failing - known, unresolved, and not new information.
• Stable - passed both times.
• New or not run - the two suites did not cover the same ground, so they are not directly comparable and the tool should say so rather than quietly averaging.

A single regression on a case someone marked as a release blocker is a stronger signal than any aggregate, and it is actionable in a way a percentage never is.

Why the aggregate persists anyway

Because it is easy to compute, easy to put on a dashboard, and it goes up when things improve. Those are properties of a metric that survives, not of a metric that is useful. A number that cannot distinguish two opposite situations is not measuring the thing you care about.

What we show instead

Comparison leads with counts by direction and an explicit verdict: safe, blocked, or inconclusive. Inconclusive is a real outcome, if the two runs did not cover the same scenarios, the honest answer is that they cannot be compared, not a number split to one decimal place.

The pass rate is still there. It is just not the headline, because it should not be the thing anyone decides on.