18 September 20266 min read
What it takes to gate a deploy on agent behaviour
Unit tests are deterministic and fast. Agent tests are neither. Here is how to gate on them anyway.
Read itAll resources
Resources · Engineering notes
Notes on testing agents that take actions
What building the detector taught us, including the parts where we got it wrong first.
16 September 20265 min read
Pass rate is the wrong number
A version that fixes two scenarios and breaks two has an identical pass rate and a completely different meaning.
Read it9 September 20264 min read
Why the transport changes what you can prove
The same agent, tested over HTTP and over WebSocket, yields different amounts of truth. Here is the part that differs.
Read it4 September 20265 min read
What twenty hand-written scripts missed
Scripted conversations are reproducible and cheap. They are also blind in a specific, predictable way.
Read it27 August 20264 min read
Not verified is not a pass
A test suite that quietly passes the checks it could not perform is worse than no suite at all.
Read it26 August 20267 min read
Beyond the LLM judge: what deterministic checks catch that scoring models miss
LLM judges are useful for tone and helpfulness. They are the wrong tool for verifying that a business rule was followed or a tool actually ran.
Read it19 August 20266 min read
Building a regression culture for AI agents
Every incident should end with a permanent test. Here is how teams turn one-off agent failures into a suite that keeps growing.
Read it18 August 20265 min read
Claimed, not executed
The most expensive agent failure is a confident sentence about something that never happened, and a transcript cannot see it.
Read it5 August 20266 min read
The five layers of agent reliability
Conversation, policies, tools, outcomes and regressions, a practical model for deciding what to evaluate and in what order.
Read it29 July 20266 min read
Which agent reliability metrics actually predict incidents
Pass rate alone hides the failures that matter. A short guide to the metrics worth putting on a dashboard.
Read it22 July 20267 min read
Policies as code: making business rules testable
Refund windows and discount ceilings should be typed rules evaluated mechanically, not paragraphs buried in a system prompt.
Read it15 July 20265 min read
Silent tool failures: when the API said no and the agent said yes
Tool errors that the agent papers over are harder to find than outright hallucinations. Here is how to surface them.
Read it8 July 20266 min read
One sentence, three broken flows: prompt changes and hidden regressions
Why small prompt edits produce distant behaviour changes, and how version comparison surfaces them before release.
Read it1 July 20265 min read
Testing escalation: the handoff is part of the product
Most escalation bugs are not about whether the agent escalated, but about what the human received when it did.
Read it24 June 20266 min read
Continuous integration for AI agents
What a sensible CI pipeline looks like when the artefact under test is a prompt, a model version and a tool set.
Read it17 June 20265 min read
What good failure evidence looks like
A failure report an engineer can act on in one sitting: transcript, trace, root cause, evaluator verdicts and a repro seed.
Read it