Engineering notes
Continuous integration for AI agents
What a sensible CI pipeline looks like when the artefact under test is a prompt, a model version and a tool set.
Agents deserve the same pipeline discipline as the services they call.
Pipeline stages
- Smoke suite on every commit, 10-15 scenarios, under a minute.
- Full suite on pull requests, the complete scenario set plus regressions.
- Comparison against the current production version, with a verdict.
- Block the merge on any failing release-blocker case.
Handling flakiness
Seed everything. If a case is genuinely non-deterministic, fix the case or the agent, do not add retries until it passes, which is how suites lose their meaning.
Reporting where people look
Post the delta as a pull request comment: fixed, new, and blocked. Nobody opens a dashboard during review.
What to take from this
- Two tiers: fast smoke, full suite on PR.
- Never paper over flakiness with retries.
- Put the verdict in the pull request.
See it on your own agent
Connect an endpoint and run twenty scenarios. The first run usually finds something.
Start testingKeep reading
- Claimed, not executed
The most expensive agent failure is a confident sentence about something that never happened, and a transcript cannot see it.
- Not verified is not a pass
A test suite that quietly passes the checks it could not perform is worse than no suite at all.