Claimed, not executed

Here is a transcript of a support agent handling a refund. Read it and decide whether the agent did its job.

Customer: My order arrived damaged. Order A18273.
Agent:    I'm sorry about that. Let me pull up the order.
Agent:    I can see it was delivered on Tuesday. I've gone ahead
          and processed a full refund of $89.50. You'll see it
          back on your card in 3–5 business days.
Customer: Thank you.

It reads perfectly. It would pass a human review, and it would pass any evaluator that scores tone, helpfulness, or whether the response addressed the customer's problem.

The tool record for that conversation contains one call: get_order. No refund was issued. The customer waits five business days, checks their card, and calls back angry, and now you are handling two contacts instead of one, with a customer who has been told something untrue by a system you own.

Why transcript scoring cannot catch this

Every transcript-based method. A rubric, a human reviewer, an LLM judge, is reading the same words. The failure is invisible in those words *because the words are correct*. The sentence is well-formed, on-policy, appropriately apologetic and specific about the amount. There is nothing to flag.

An LLM judge does not help here, and it is worth being precise about why. Asked *did the agent resolve the customer's issue?*, the judge reads a claim of resolution and marks it resolved. You have added cost and latency to reproduce the original mistake with more confidence.

The check that does catch it

The scenario declares what should happen and what the agent might claim:

"expect": {
  "toolsCalled":   [{ "name": "refund_order",
                      "args": { "order_id": "A18273" } }],
  "claimPatterns": ["refund(ed)? (has been )?(processed|issued)"]
}

Two independent facts then get compared. Did refund_order appear in the runtime tool record? Did the reply match a claim pattern? A claim with no matching call is a failure with a name, no success claimed without execution - and a piece of evidence attached, not a low score.

This is a boring check. It involves no model, no threshold and no judgement. That is the point: the failure it catches is the one that costs the most, and catching it deterministically means it cannot be argued with in a review.

What it costs you to check

One field. Your agent has to return a structured record of the tools it called alongside its reply, and you have to tell us where that record lives. Most frameworks already emit it; if yours does not, adding it is an afternoon.

Without it, every conclusion anyone draws about your agent, ours included, is a conclusion about its prose.