Connect an agent, and know what the verdicts mean

ActionProof holds conversations with your agent and then checks what it executed against what it claimed. To do that it needs two things from you: somewhere to send a customer message, and somewhere to read the tool calls that message produced.

1. Create a workspace

Signing up creates a workspace and installs a starter suite of twenty scenarios covering refunds, cancellations, shipping, escalation and privacy. You can run them before writing anything of your own.

2. Connect an agent

Under Agents → Add agent, give the endpoint, the authentication and two mappings: how to phrase a request, and where in the response the reply and the tool record live.

{
  "endpoint": "https://api.example.com/v1/support/chat",
  "method": "POST",
  "auth": { "type": "bearer" },
  "requestMapping": {
    "message": "{{input}}",
    "session": "{{session_id}}"
  },
  "responseMapping": {
    "reply":   "$.output.text",
    "tools":   "$.output.tool_calls",
    "results": "$.output.tool_results"
  }
}

3. Test the connection

Test connection sends one real message and reports what came back: the status code, the latency, and whether the tools path resolved to an array. A path that resolves but is empty on a greeting is healthy. That turn simply made no calls. A path that does not resolve at all fails the check, because it would silently report every agent as perfect.

4. Run the suite

Run tests queues one item per scenario. A worker claims them, holds each conversation to its natural end, and writes the transcript, the tool record and every check to the database. A twenty-scenario suite against a local agent finishes in well under a minute; against a hosted agent it is bounded by your own latency.

A connector is the stored description of how to reach one agent. The engine supports two transports today.

Request mapping

The request body is your own shape with placeholders substituted. A placeholder that is the entire value keeps its native type; embedded ones stringify.

Response mapping

Tool records vary. We read name or tool or function for the name, and arguments or args or input for the arguments, so most shapes work unchanged.

Session handling

Multi-turn conversations need continuity. Declare where the session identifier belongs, in the body, in a header, or in the URL path.

"session": { "mode": "body",   "field": "session" }
"session": { "mode": "header", "field": "X-Session-Id" }
"session": { "mode": "url",    "field": "session_id" }

Authentication

Four schemes: none, bearer, api_key with a header name of your choosing, and basic. The credential itself is encrypted at rest and the connector row stores only a reference to it. It is decrypted in the worker process at the moment a request is made, and never written to a log or returned by the API.

A third transport, WhatsApp, works differently enough to have its own section below.

WebSocket

A socket gives a stream of frames with no inherent notion of *the agent has finished replying*. Every agent invents its own signal, so the signal is declared rather than guessed. Four conventions cover what agents actually do:

{
  "transport": "ws",
  "endpoint": "wss://agent.example.com/chat",
  "ws": {
    "hello":        { "type": "hello", "token": "{{session_id}}" },
    "send":         { "type": "message", "text": "{{input}}" },
    "typePath":     "$.type",
    "textDelta":    { "type": "chunk", "path": "$.text" },
    "toolCall":     { "type": "tool_call" },
    "turnComplete": { "mode": "type", "type": "done" }
  }
}

WhatsApp is structurally different from the other transports. HTTP and WebSocket hand the reply back on the connection we sent on; WhatsApp does not. We post to Meta's Graph API, Meta acknowledges, and the agent's actual reply arrives later as a webhook.

We are the customer, so we bring the phone

You give us one thing: the WhatsApp number your agent answers on. We message it from our own number, exactly as a customer would, and your agent's replies come back to us.

There is nothing to configure on your side. No access token to hand over, no webhook to point anywhere.

The one wrinkle: the first message

Meta will not deliver free-form text to a number that has not messaged us in the last 24 hours, so the opening turn of each conversation goes out as a pre-approved template. Your agent replies, the window opens, and the rest of the conversation is ordinary text.

In practice this constrains the first line a scenario can say, not the scenario. A template like Hi, I have a question about my order: {{1}} carries most openings unchanged.

If you would rather use your own number

Supported, and occasionally the right call. You want the test to originate from a specific number, or you want to exercise your own outbound flows. Then you supply a phone number id and an access token from a separate Meta app, never the one serving production, and point that app's webhook at https://actionproof.ai/api/webhooks/whatsapp.

Constraints that shape a suite

• The 24-hour window. Outside it, only pre-approved templates can open a conversation. A suite that opens with free text will fail with Meta error 131047.
• Rate limits. You cannot fire a hundred scenarios at one number and expect them all through. Pace the suite or use more numbers.
• One number, one conversation at a time. Replies correlate by the sending number, so scenarios against the same agent run in sequence rather than in parallel.
• Reputation. Meta quality-rates the number that sends. We manage a pool so one noisy suite cannot degrade delivery for everyone.

The shape

The engine never touches audio. A separate bridge owns speech entirely and talks to us over the WebSocket protocol we already support:

[worker] ──WebSocket(text)──> [bridge] ──AudioSocket(PCM)──> [Asterisk] ──SIP──> [agent]

That seam is not new. The heard frame, where an agent reports its own transcription, was built before any telephony work, so the evaluator that separates *the agent did the wrong thing* from *the agent was told the wrong thing* works over a phone call unchanged.

Latency, attributed

A single latency number cannot diagnose a slow voice agent, because it conflates three independent things. The bridge emits three marks instead, and the engine subtracts them.

A system with fast inference and lazy endpointing scores identically to the reverse on a single number, and the operator cannot tell which to fix. This is tested: a 960 ms turn resolves to 407 ms of endpointing, 299 ms of thinking and 252 ms of speech.

Tested end to end

• The timing model and its attribution, end to end against a bridge that emits the marks.
• The transcription evaluator, which already attributes failures to the speech pipeline rather than the agent's reasoning.
• Transcript mode, which is what a voice agent gets. The protocol carries audio, not tool calls.
• The bridge's AudioSocket framing and the Asterisk dialplan.

Not built

• Voice activity detection worth the name. The bridge uses a naive energy gate; endpointing quality decides whether any measurement means anything.
• Asterisk deployment, a SIP trunk and numbers.
• A SIP trunk and numbers, so a scenario can dial a real phone.

Every verdict carries the level of evidence it rests on. This is the core of the product and the reason a check can come back neither pass nor fail.

Not verified is never a pass

When a check cannot be answered with the evidence at hand, it is recorded as not_verified and rendered in neutral grey, not green. A test suite that quietly passes the checks it could not perform is worse than no suite, because it manufactures confidence.

The clearest example: *final state matches expected*. Confirming the account really ended up in the right state requires observing the backend, which at the runtime-record level we do not do. That check reports not_verified on every run, and it will keep reporting it until there is a ledger to read.

Checks are deterministic. No model is asked for a judgement anywhere in this path. A scoring model that marks its own homework is the thing we are replacing.

Tool execution

Transcription

For voice, the agent tells us what it heard. Because our synthetic customer wrote the line, we hold the ground truth for one side of the conversation, so when a failure follows a mistranscription, the failure is attributed to the speech pipeline rather than to the agent's reasoning. Without that split a voice suite blames the agent for words it was never given.

A scenario is split three ways on purpose: what drives the conversation, what the target environment must already contain, and what the evaluator consumes. Only the third part produces verdicts.

{
  "ref": "SC-142",
  "name": "Refund damaged item",
  "category": "Refunds",
  "severity": "Critical",
  "persona": { "archetype": "Frustrated customer" },
  "context": "Order A18273 arrived damaged two days ago.",
  "goal": "Obtain a refund for the damaged product.",
  "maxTurns": 12,
  "preconditions": {
    "orders": [{ "id": "A18273", "total": 89.5, "status": "delivered" }]
  },
  "expect": {
    "toolsCalled": [
      { "name": "refund_order",
        "args": { "order_id": "A18273" },
        "match": "subset" }
    ],
    "claimPatterns": ["refund(ed)? (has been )?(processed|issued)"]
  },
  "forbid": { "toolsCalled": ["check_shipping"] }
}

Scripted or generated customers

A scenario with a script runs canned turns, fast, free and byte-identical between runs, which makes it the right choice for a regression gate. A scenario without one is driven by a model playing the persona, which goes off-script the way real customers do.

Guards

• Production refusal. An agent marked Production cannot be run, checked both when the run is queued and again when an item is claimed.
• Monthly spend cap. A generated customer costs model tokens. Every workspace has a cap; runs beyond it stop rather than continuing silently.
• Leases. A worker that dies mid-conversation loses one item, which returns to the queue and is retried.

Not built yet

Stated plainly so nobody discovers it during an evaluation: the independent ledger, server-sent-events transport, browser automation, scenario generation from a capability description, the policy engine, version comparison, and a composite reliability score. Two evaluators are implemented, tool execution and transcription, not nine.