All notes

Engineering notes

What good failure evidence looks like

A failure report an engineer can act on in one sitting: transcript, trace, root cause, evaluator verdicts and a repro seed.

17 June 20265 min read

Triage time is the metric nobody tracks and everybody feels.

The five required artefacts

  • Full transcript with turn numbers.
  • Tool trace with arguments, results and timing.
  • Expected vs actual, rendered as a diff.
  • Per-evaluator verdicts with reasoning.
  • A seed that reproduces the run exactly.

Point at the turn

The report should name the exact turn where behaviour diverged. Scrolling a 40-turn transcript looking for the moment is where triage time goes.

One click to permanence

From the evidence view, promoting the case into the regression suite should take one action and keep the seed attached.

What to take from this

  • Evidence, not scores, is what gets bugs fixed.
  • Name the diverging turn explicitly.
  • Keep the repro seed with the case forever.

See it on your own agent

Connect an endpoint and run twenty scenarios. The first run usually finds something.

Start testing

Keep reading

  • Claimed, not executed

    The most expensive agent failure is a confident sentence about something that never happened, and a transcript cannot see it.

  • Not verified is not a pass

    A test suite that quietly passes the checks it could not perform is worse than no suite at all.