Building a regression culture for AI agents

Software teams solved this decades ago: a bug is not fixed until there is a test that fails without the fix. Agent teams are still relearning it.

Why agents regress so easily

A prompt is a global variable. Adjusting one sentence to fix escalation behaviour can change refund behaviour three flows away. Model upgrades do the same thing at a larger scale.

Without a suite, every release is a fresh gamble on behaviour you already paid to fix once.

The promotion loop

Make the path from incident to permanent test one click, or it will not happen under deadline pressure.

• A failure is found in testing or production.
• It is promoted to a regression case with its seed and expected behaviour.
• It runs on every future version, producing a PASS/FAIL column per version.
• Critical cases are marked as release blockers.

Deciding what blocks a release

Not everything should block. Money movement, data deletion, compliance disclosures and escalation failures are blockers. Tone drift and phrasing changes are warnings.

Write that classification down once and apply it mechanically, so nobody negotiates it at 6pm on a Friday.

Keeping the suite healthy

Review the suite quarterly. Retire cases whose flows no longer exist, and watch for cases that pass for the wrong reason after a refactor.

What to take from this

• Promotion from failure to test must be frictionless.
• Classify blockers before you need the answer.
• A suite that never gets pruned stops being trusted.