Reliability infrastructure for customer-facing AI
Ship AI agents you can trust.
Automatically test your customer support agents across thousands of realistic conversations. Catch hallucinations, broken workflows, policy violations and failed actions before your customers do.
No production traffic required · HTTP, WebSocket, WhatsApp and Voice
Reliability Score
Tests
1,284
Passed
1,219
Failed
65
Task Completion
Policy Compliance
Tool Accuracy
Hallucination-Free
| F-2291 | Refund damaged item | Critical | FAIL |
| F-2288 | Cancel subscription | High | FAIL |
| F-2284 | Shipping estimate | Medium | FAIL |
Built for teams shipping AI agents across
Works with API-based agents today. Voice and production monitoring are coming next.
The core problem
Your agent said it worked. Did it actually?
Traditional AI evaluations judge the text response. A fluent, polite, on-policy answer can still describe an action that never happened.
What the customer sees
Customer
“My order arrived damaged. Can I get a refund?”
Acme Support Agent
“I've processed your €49.99 refund. You should see it within 3–5 business days.”
What actually happened
Root cause: The agent claimed success without executing the refund.
ActionProof verifies the conversation, the actions and the actual business outcome.
How it works
Test the entire agent journey.
From connection to root cause in four steps. Everything runs against your development, staging or dedicated test environment.
Connect your agent
Connect any API-based support agent.
No SDK required for basic testing.
Generate realistic scenarios
Generate hundreds of synthetic customer conversations from policies, knowledge, workflows, historical failures and edge cases.
Run conversations
Synthetic customers interact with the agent across realistic multi-turn conversations, not single-turn prompts.
Find what broke
Every run is evaluated across conversation, policies, tool calls, outcomes, latency, cost and regressions.
Five layers of reliability
Reliability is more than response quality.
Each layer is evaluated independently, so a good answer can still fail on the action it claims to have taken.
Conversation
Did the agent understand the customer and respond correctly?
Policies
Did the agent follow the rules it must never violate?
- Authentication
- Refund limits
- Privacy
- Escalation rules
- Compliance requirements
Tools
Did the agent call the right tools with the right arguments?
Expected
refund_order( order_id="A18273", amount=49.99 )
Actual
No call detected.
Outcomes
Did the customer's request actually get completed?
Regressions
Did a new model, prompt or workflow change break something that worked before?
Failure explorer
Every failure comes with evidence.
Transcript, tool trace, evaluation checks and root cause in one view, so engineers debug the agent, not the report.
My order A18273 arrived damaged.
I'm sorry about that. Let me verify your order.
I've processed your €49.99 refund. It will arrive in 3–5 business days.
Great, thank you!
expected: refund_order(order_id="A18273", amount=49.99)
actual: - no tool call emitted
Agent asserted completion of a refund that was never executed. Customer impact: unfulfilled refund.
Regression testing
Fix it once. Keep it fixed.
Every confirmed failure becomes a permanent test. Model swaps, prompt edits and workflow changes are checked against your entire regression suite before release.
- Turn any failure into a regression test in one click
- Run the suite on every agent version
- Block releases when critical tests fail
- Track fix history version by version
| Test | Scenario | v1.41 | v1.42 | v1.43 |
|---|---|---|---|---|
| RT-014 | Refund damaged item | FAIL | FAIL | PASS |
| RT-013 | Cancel subscription mid-cycle | PASS | FAIL | PASS |
| RT-011 | Escalate after 2 failed attempts | PASS | PASS | PASS |
| RT-008 | Refuse refund over policy limit | PASS | PASS | PASS |
Fixed failures
18
New regressions
12
Behavior changes
31
Version comparison
Know exactly what a change did.
Compare any two agent versions across reliability, completion, tool accuracy, latency and cost. See what got fixed, what regressed and what silently changed behavior.
- Model upgrades: GPT-4o → Claude, Llama or in-house
- Prompt and workflow revisions
- Tool and API changes
- Knowledge base updates
Policies as code
Encode the rules your agent must never break.
Define authentication, refund limits, privacy and escalation rules once. Every scenario is checked against them, and violations are reported as policy failures with the exact rule that was broken.
Refund & Returns Policy
12 rules · applied to 384 scenarios
Under the hood
Not just an LLM judge.
Language models are good at grading language. They are not evidence of execution. We combine deterministic verification with model-based evaluation, and we tell you which is which.
Deterministic checks
Tool calls, arguments and ordering are checked against your agent runtime’s own record of what it executed. No opinion involved.
assert tool_called("refund_order")
assert args.amount == order.total
assert not claimed_before(call)Structured evaluators
Weighted rubrics per dimension, with thresholds you control, rather than a single opaque score.
Reproducible runs
Fixed seeds, pinned scenario versions and recorded traces mean a failure can be replayed exactly as it happened.
- Versioned scenarios and evaluators
- Full request / response traces
- Deterministic replay of any run
- Exportable evidence for audits
In your pipeline
Fail the build, not the customer.
Run the suite on every pull request. When a case that must never break, breaks, the build stops, and the reason is a named check with the tool record behind it, not a score.
- name: Run the regression suite
env:
ACTIONPROOF_API_KEY: ${{ secrets.ACTIONPROOF_API_KEY }}
run: |
curl -fsSL https://actionproof.ai/ci/actionproof.mjs \
-o actionproof.mjs
node actionproof.mjs run \
--agent my-support-agent \
--suite regression \
--wait
# BLOCKING SC-142 Refund damaged item
# refund_order never appears in the tool record
# gate: blocked -> exit 1The verdict is ours, not your shell script
The API returns an explicit gate, pass, warn or blocked. If every team derives it from counts themselves, every team derives it slightly differently, and eventually a blocking regression ships because of an off-by-one.
Production is refused
A run executes real tool calls. An agent marked Production is rejected by the API, because a pipeline pointed at it would issue real refunds on every pull request.
Or skip polling entirely
Add a signed webhook and we post the result when the run finishes, enough to gate a deploy on, because the receiver can verify we sent it.
Use cases
Wherever an agent takes real actions.
If your agent can move money, change accounts or promise an outcome, it needs verification, not vibes.
E-commerce
Refunds, returns, order changes and shipping exceptions verified end to end.
Fintech
Identity checks, disputes and transaction actions tested against compliance rules.
Telecom & subscriptions
Cancellations, plan changes and retention flows that must actually execute.
SaaS support
Account provisioning, billing changes and escalation to human agents.
Insurance
Claims intake and policy questions with strict disclosure requirements.
Marketplaces
Two-sided disputes where the wrong action costs money on both sides.
Built for the whole team
One source of truth for agent quality.
AI engineers
Debug agent behavior with full traces instead of screenshots from support tickets.
- Root-cause failures
- Compare model versions
- CI test runs
QA & test teams
Replace manual spot-checking with thousands of automated conversations per release.
- Scenario coverage
- Regression suites
- Release gates
Support leaders
Know your automation rate is real, and that customers actually got what they were promised.
- Reliability score
- Failure trends
- Risk by category
Compliance & risk
Evidence that policy rules were enforced, with exportable audit trails.
- Policy coverage
- Violation reports
- Reproducible runs
Integrations
Works with the stack you already have.
Framework-agnostic by design. If your agent can be reached over HTTP, it can be tested, and results can be pushed into CI, ticketing and alerting.
Beyond text
Voice today, production traffic next.
The same verification model, applied to spoken conversations now and to live traffic after that.
Voice agent testing
A synthetic caller speaks your scenarios. Endpointing, time to first audio and speech duration are reported separately, and a mishearing is attributed to the speech pipeline rather than charged to the agent.
Production monitoring
Sample live conversations, detect claimed-but-not-executed actions in real traffic and alert before customers complain.
Security & data handling
Synthetic test data by default, encryption in transit and at rest, scoped API keys and role-based workspace access.
Pricing
Priced by conversations tested, not seats.
Every plan includes the full five-layer evaluation. Start free, upgrade when your suite grows.
Starter
Evaluate a single agent with a small scenario suite.
- 1 agent
- 100 test conversations / month
- Core evaluators
- Failure explorer
- 7-day run history
Team
Most popularFor teams shipping agent changes every week.
- 5 agents
- 10,000 test conversations / month
- Scenario generation
- Regression suites & version compare
- Policies and custom evaluators
- CI integration and webhooks
Enterprise
For regulated and high-volume support operations.
- Unlimited agents
- Custom volume
- SSO and audit logs
- Private deployment options
- Voice testing early access
- Dedicated support engineer
FAQ
Questions, answered.
No. ActionProof drives synthetic conversations against a development, staging or dedicated test environment. Production monitoring is a separate capability that is coming later.
Model-based judging only grades the text. We additionally verify tool calls, their arguments and the resulting business outcome deterministically, so an agent cannot pass by sounding confident.
Yes. Any agent reachable over HTTP works, LangChain, LangGraph, CrewAI, vendor platforms or a fully custom stack. You map the request and response fields once during setup.
Every workspace starts with twenty scenarios covering refunds, cancellations, shipping, escalation and privacy. You write your own alongside them, and any failure can be promoted into a permanent regression case. Generating scenarios from your own policies and knowledge base is planned, not built.
Scenarios use synthetic customers and synthetic records by default. Test runs point at sandboxed tools and APIs, so no real refunds, cancellations or account changes occur.
Yes. Trigger suites from GitHub Actions or any CI system, gate releases on critical regression tests and receive results via webhook or Slack.
Yes. A synthetic caller speaks the same scenarios you run over chat, and we transcribe what the agent says back. Latency is reported as three separate measures, endpointing delay, time to first audio and speech duration, because one number cannot tell you whether to tune the voice detection or the model. Agents reachable over our voice bridge work today; dialling a phone number needs a SIP trunk, which is in progress.
Find out what your agent is really doing.
Connect an agent, generate a scenario suite and get your first reliability report in under an hour.