How to keep an AI agent that writes production code under control

Why is the model not the guarantee?

A model can write correct code ten times and wrong code the eleventh, in exactly the same confident tone. If production quality depends on something you cannot predict, you have no guarantee. It has to depend on checks that run whatever the model wrote.

The same logic applies to any part you do not control from the inside. You do not trust a subcontractor because they are good. You trust them because their work goes through acceptance. NIST's AI Risk Management Framework (AI RMF 1.0) makes the same point in general terms: AI risk is managed through processes around the system, across its whole life.

In Planner I applied that literally. The system takes a specification and delivers a working application, passing through architecture, implementation with tests written before the code, quality checks, an audit and deployment. None of those steps depends on how clever the model happens to be that day.

Models change every few months. The mechanisms stay. That is why I put my effort there.

How do you write completion criteria that leave no room for interpretation?

A completion criterion leaves no room for interpretation when a machine can check it and answer yes or no. “The page loads fast” is a wish. “The form test passes and the page returns 200” is a criterion. Criteria are written before the code, and a task is finished only when all of them pass.

The reason is practical. An agent will say “done” just as easily whether it is done or not. If the agent judges its own “done”, you have let the student mark their own exam.

So the work moves earlier. Before implementation starts, every task has a list of checks that must turn green. This is TDD applied to agents: the tests are the specification, not an afterthought.

What makes a criterion good

A good criterion has three properties. A machine can check it. It describes the result, not the method. And code that does the wrong thing cannot satisfy it.

The third is the hardest. If the criterion says “the test passes”, the agent has an easy way out: change the test. That is why one rule of the system is that it never loosens a test, a type or a rule just to get green. If a criterion is wrong, it is changed openly and the change is recorded, not slipped quietly into the implementation.

How do you verify every finding before it is accepted?

Every finding from a review agent, whether a security hole or a bug, is checked again against the current code before it is accepted. If a failing test reproduces it, the finding is real and gets fixed. If it cannot be reproduced, it is rejected with a reason.

Agents that do code review, pentesting or QA exploration produce findings, and some of them are wrong. They point at code that has since changed, misread a pattern, or suggest something already covered elsewhere. Accept them all and you fill the project with changes nobody needed, and sometimes add new bugs.

In Agent Studio, specialised agents run in parallel, each in its own area: code review, security, pentest, QA, UX, SEO/AEO. A separate verification step then asks not “does this sound reasonable?” but “can I prove it?”. A finding is either proven with a failing test or disproven by an active check.

Every answer becomes a rule

What is answered once becomes a rule. If an agent proposes something and verification rejects it, the rejection is recorded so it is not proposed again in the next round. Without that, the same false suggestions come back every cycle and eat time.

Verification also keeps the code honest in another way: each finding is checked against the code as it is now, not a stale picture of it. A finding that was right three changes ago may already be fixed.

What does automatic rollback do when a deployment fails?

When a deployment fails, automatic rollback puts the previous working version back without waiting for a person. The service returns to a known good state, and the failure is recorded so it can be fixed calmly rather than under pressure. The damage is limited to the time the system takes to notice the failure.

Rollback is the last line of defence, which is why it must not depend on anyone. Tests and verification lower the chance that bad code reaches production. They do not make it zero. A production environment always has something a test environment lacks: real data, real load, a setting that differs.

Two things make rollback work. First, the previous version stays available until the new one is confirmed. Second, the decision comes from a health check after deployment, not from someone's mood. And the final check is never delivered red: if something fails before deployment, deployment does not start.

How much human involvement is left?

In Planner a person is needed in two places: in the feedback rounds, where the owner writes what they want and the system applies it and shows the result again, and in approving the deployment. Every step in between runs without waiting for them, and the decision about what reaches production always stays with a person.

Approval at deployment is deliberate. It is the point where a wrong decision leaves the lab and touches real users, and I want a pair of human eyes there. Everywhere else, the checking is done by mechanisms that do not get tired or bored.

The system also saves its state at every step. If the process dies, it continues from where it stopped instead of starting over. One project runs on one machine at a time, with a lock and limits on memory and load, so two tasks never collide.

How do you start building the same mechanisms?

Start with the smallest piece that has value: one machine-checkable completion criterion for each task. Then add verification of findings, and last the automatic rollback. Each step works on its own, so you do not have to build the whole system before you see a benefit.

The steps, in order:

1. For every task, write the completion criteria as checks that pass or fail, before the agent starts.
2. Forbid loosening a test, type or rule to get green. Changing a criterion is done in the open.
3. For every finding an agent produces, ask for proof in the current code: a failing test or a documented refutation.
4. Keep a record of rejected findings so they do not return.
5. Tie deployment to a health check and an automatic return to the previous version.
6. Keep human approval at the point where the change touches real users.

If you need something like this in your team, you can talk to me. The wider context of my work with AI agents is on the home page, and a comparable example of a system that decides on its own is blocking scanners by their behaviour.