Engineering notes
What good failure evidence looks like
A failure report an engineer can act on in one sitting: transcript, trace, root cause, evaluator verdicts and a repro seed.
Triage time is the metric nobody tracks and everybody feels.
The five required artefacts
- Full transcript with turn numbers.
- Tool trace with arguments, results and timing.
- Expected vs actual, rendered as a diff.
- Per-evaluator verdicts with reasoning.
- A seed that reproduces the run exactly.
Point at the turn
The report should name the exact turn where behaviour diverged. Scrolling a 40-turn transcript looking for the moment is where triage time goes.
One click to permanence
From the evidence view, promoting the case into the regression suite should take one action and keep the seed attached.
What to take from this
- Evidence, not scores, is what gets bugs fixed.
- Name the diverging turn explicitly.
- Keep the repro seed with the case forever.
See it on your own agent
Connect an endpoint and run twenty scenarios. The first run usually finds something.
Start testingKeep reading
- Claimed, not executed
The most expensive agent failure is a confident sentence about something that never happened, and a transcript cannot see it.
- Not verified is not a pass
A test suite that quietly passes the checks it could not perform is worse than no suite at all.