Engineering notes
Why the transport changes what you can prove
The same agent, tested over HTTP and over WebSocket, yields different amounts of truth. Here is the part that differs.
A request/response API returns a turn as one object: the reply text, and the tools called during that turn. It does not say which came first.
Usually that does not matter. Occasionally it is the whole question. An agent that says *your refund is processed* and then calls refund_order has done something different from one that calls it first. The customer was told a thing that was not yet true, and if the call had failed they would never have learned otherwise.
What we do about it
We do not guess. Over HTTP, ordering within a turn is recorded as unknown, and a check that depends on it reports not_verified. Over a streaming transport the frames arrive in order, so we know whether the call preceded the sentence, and the same check produces a real verdict.
The difference is measurable. Running a deliberately clean agent configured to call its tools *after* replying: over HTTP, twenty scenarios passed and ordering was known for none of the twenty-eight calls. Over WebSocket, the same agent produced three failures, and ordering was known for all twenty-eight.
Same agent. Same scenarios. Different evidence.
Nothing about the agent changed between those runs. What changed is how much we could see, and therefore how much we were willing to assert.
The practical advice
If your agent already speaks WebSocket, test it that way even if your production traffic is HTTP. You get strictly more evidence for the same effort. If it does not, HTTP is fine and the report will simply be explicit about the questions it left open.
That explicitness is the feature. A tool that reports the same verdict regardless of what it could observe is telling you about its own defaults, not about your agent.
See it on your own agent
Connect an endpoint and run twenty scenarios. The first run usually finds something.
Start testingKeep reading
- Claimed, not executed
The most expensive agent failure is a confident sentence about something that never happened, and a transcript cannot see it.
- Not verified is not a pass
A test suite that quietly passes the checks it could not perform is worse than no suite at all.