Which agent reliability metrics actually predict incidents

A 94% pass rate sounds excellent until you learn the 6% are all refunds.

Weight by blast radius

Weight failures by what they touch. A tone failure and a money-movement failure should never contribute equally to a headline number.

Metrics worth tracking

• Claim-execution mismatch rate, claims not backed by a tool call.
• Policy violation rate, split by rule severity.
• Silent tool failure rate, errors the agent smoothed over.
• Escalation correctness, handed off when it should, with context.
• Regression count per release.
• Median triage time from failure to root cause.

Trend over snapshot

A single run tells you about that run. Direction across versions tells you whether your process is working.

What to take from this

• Unweighted pass rate is a vanity metric.
• Track execution mismatch separately, it is the leading incident indicator.
• Judge releases on deltas, not absolutes.