Not verified is not a pass

Most test tooling has two outcomes. Something passed, or something failed. That binary is fine when the test either ran or crashed. It is actively harmful when the test *ran but could not see enough to decide*.

Agent testing is full of the second case. We can read the tools an agent reported calling. We cannot, from that alone, confirm the customer's account actually ended up in the right state. That would mean observing the backend, and an agent's own record is not an independent witness to its own effects.

The dishonest version

The tempting move is to drop the check, or to define it as whatever we *can* observe and call that a pass. Both produce the same artefact: a green suite that has quietly stopped asking the hardest question. Somebody ships on that green.

We built this wrong once. An early build rendered not_verified with the same red styling as a failure, and a later screen displayed *sandbox verified* next to a state we had no sandbox to verify. Both were caught in review, and both were exactly the dishonesty the evidence model exists to prevent, which is a useful reminder that the failure mode is not malice, it is a default.

Three levels, stated on every verdict

• Transcript - the words only. Supports nothing about actions.
• Runtime record - the agent's own report of what it called. Supports which tools ran and with what arguments.
• Independent ledger - the same facts observed from outside the agent. Supports claims about real effects.

A check that needs a level you have not reached returns not_verified and renders in neutral grey. It is not a failure, nothing went wrong. It is a statement about the limits of the evidence, carried all the way to the screen instead of being rounded to the nearest convenient colour.

What this buys you

When your suite is green, you know precisely what that green covers. When a regulator, a customer or your own CTO asks *how do you know*, the answer is a specific artefact rather than a number nobody can derive.

And when we tell you something passed, you can believe it, because you have seen us decline to say so elsewhere.