Engineering notes
One sentence, three broken flows: prompt changes and hidden regressions
Why small prompt edits produce distant behaviour changes, and how version comparison surfaces them before release.
Prompts have no module boundaries. Every instruction is global, and the model blends them in ways no diff tool can show you.
The coupling problem
Adding "be more concise" can remove a required disclosure. Adding "always confirm the order number" can make the agent stall on customers who already provided it.
Compare behaviour, not text
A prompt diff tells you what you typed. A behaviour diff tells you what changed: failures fixed, failures introduced, and behaviour shifts like escalation rate or tool usage that no pass rate captures.
Release verdicts
The output of a comparison should be a plain call, safe to release, or blocked, with the blocking cases listed. Anything more ambiguous gets rationalised away.
What to take from this
- Prompt edits have non-local effects by design.
- Compare behaviour across identical seeded scenarios.
- Ship on an explicit verdict, not on a feeling.
See it on your own agent
Connect an endpoint and run twenty scenarios. The first run usually finds something.
Start testingKeep reading
- Claimed, not executed
The most expensive agent failure is a confident sentence about something that never happened, and a transcript cannot see it.
- Not verified is not a pass
A test suite that quietly passes the checks it could not perform is worse than no suite at all.