What it takes to gate a deploy on agent behaviour

Putting agent tests in a pipeline sounds like putting any other test in a pipeline, and it is not, for three reasons that all have the same root: the thing under test is not deterministic and does not run in-process.

They take minutes, not milliseconds

A twenty-scenario suite holds twenty multi-turn conversations with a live agent. Against a hosted agent that is bounded by their latency, not ours. Any design that assumes a test run finishes inside a request is wrong, which is why a run is queued work with a status endpoint rather than a synchronous call.

The practical consequence: gate pull requests on the regression subset, and run the full suite on a schedule. A gate slower than the deploy it guards gets skipped under pressure, and a gate people skip is worse than none. It provides cover without providing safety.

They can flake, unless you choose otherwise

A model-driven customer produces a different conversation every time. That is exactly what you want when hunting unknown failures, and exactly what you do not want in a blocking check, where a red build must mean the agent changed.

So the gate runs scripted scenarios, byte-identical between runs. We measured this: sixty generated conversations against the same agent produced zero verdict flips, which is reassuring, but the scripted path removes the question entirely.

They can cost real money, in two directions

Generated customers spend model tokens. More importantly, the agent under test executes real tool calls, so a pipeline pointed at production would issue real refunds on every pull request. Our API refuses an agent marked Production with a 409, and that refusal is not configurable.

Where CI usually goes wrong

• Blocking on everything. The suite finds a genuine but minor issue, the build goes red, someone adds continue-on-error, and the gate is gone.
• Blocking on unanswerable checks. A check that reports not_verified is not evidence of a defect. Failing a build on it teaches people the gate is noise.
• Deriving the verdict in a shell script. Everyone does it slightly differently and eventually someone gets it wrong silently. We return an explicit gate field for this reason.
• No version label. A failure you cannot attribute to a build is an argument, not a finding.

What good looks like

A regression suite of cases that were each real failures once, marked blocking only where a customer would notice, run on every pull request against staging, with the build failing on a single non-zero exit code. Everything else on a schedule, where a surprise is welcome.