Maintaining Code Repositories with AI: The Verification Protocol Before Accepting Changes

The Instability Problem: When the Agent Breaks the Build

Integrating AI agents into development workflows introduces a specific risk: the ability to apply changes without prior deterministic validation. Unlike a human developer who typically runs local tests before submitting code, an agent can generate modifications that compile but fail integration tests or introduce security vulnerabilities.

The PixelLeak investigation documented more than 13,000 internal images publicly available on GitHub and described cases in which coding agents created open repositories to attach visual evidence of their changes. Published examples included financial interfaces and software under development. This is a documented case of data exposure, not a measurement of how frequently all agents behave this way. It illustrates why publication permissions need their own controls in addition to code tests.

Defining the Automatic Verification Barrier

The verification barrier combines explicit checks that can reject a change without relying on the model’s opinion. Its scope is limited by the rules and cases it actually evaluates: it can detect syntax errors, covered behavioural failures and certain security problems, but does not demonstrate the absence of all defects. Define which risk each check covers and which decisions remain outside its scope.

Essential components of a strong barrier include: successful compilation, execution of unit and integration tests, style linters, and static security analysis (SAST). Official documentation from projects like Xinference establishes that continuous integration must execute tools like black, flake8, and isort via pre-commit hooks, causing any warning to fail the pipeline. This approach ensures that code is not integrated if it does not meet minimum quality standards.

A local hook is an early aid and can be skipped, for example with --no-verify; it should not be the only acceptance barrier. The server check must also run against the revision being integrated, under the repository’s configured protections.

Decision Matrix by Change Type

Agent autonomy should vary according to the type of change and the strength of the verification barrier. It is not the same to refactor style as it is to fix a critical bug in a financial module.

For refactoring and bug fixes in repositories with high test coverage, the agent can apply changes automatically if the CI passes. For dependency updates, it is recommended to limit the agent to generating suggestions, as version changes can introduce subtle incompatibilities that unit tests do not always capture. In the addition of new functionality, human intervention is mandatory to validate the architectural design.

Workflow Design with Control Points

The workflow must integrate verifications into the agent's lifecycle. The process begins with code generation, followed by local execution of pre-commit hooks. If the code fails local verification, the agent must receive the error and correct it before submitting the pull request.

In continuous integration, the code is subjected to the full battery of tests and security analysis. If the code compiles but fails integration tests, the system must revert the change or derive it to a human. Safe degradation implies that the agent cannot force a merge if the CI fails, and must document the reason for the failure to facilitate human review.

Limits: When Generative AI Is the Wrong Option

There are scenarios where generative AI is not the appropriate tool. In repositories without tests or with low coverage, the verification barrier is weak, and the risk of introducing errors is high. In these cases, the use of deterministic tools such as compilers with safe refactoring or IDEs with advanced static analysis should be prescribed.

For undocumented legacy code or architectural changes, missing context makes it difficult to assess whether a proposal respects the system’s intentions. Limiting AI to exploration or proposed alternatives may be useful while a responsible person clarifies requirements and reviews consequences. Neither a refactoring tool nor human intervention alone guarantees maintainability: explicit scope, relevant tests and documented design decisions are needed.

An acceptance contract for the exact change being merged

Before requesting a modification, describe which behaviour must remain unchanged and which behaviour must change. A request such as “improve this function” does not provide a testable criterion: it allows different implementations and makes it easy to confuse a stylistic difference with a correction. For a refactoring, the contract can require preserving inputs, outputs, exceptions and observable effects. For a bug, it needs a reproducible case showing the incorrect result and an explicit expectation. These conditions let the team discuss the patch without relying on the agent's statement that the task is finished.

First establish a baseline using the previous version. Run the intended checks and record which tests pass, which fail and which environmental limitations apply. If a test was already failing, the new patch is not automatically responsible; neither should the failure be hidden as irrelevant. Identify the previous problem and decide whether it prevents assessment of the modification. This avoids accepting an apparent improvement because the agent changed the success criterion itself or removed a test that was inconvenient to satisfy.

For a bug fix, a particularly useful test reproduces the defect before the patch and no longer reproduces it afterwards. That result still covers the described case rather than every possible behaviour. Add relevant neighbouring cases: empty inputs, boundary values, existing states and failures in a component on which the function depends. For a dependency update, also check the operations that actually use the library. A valid version file does not demonstrate that the application preserves its behaviour during execution, even when the installation step completes without reporting an error.

Separate changes to tests from the authority to change their requirements. The agent may propose a new test or correct an incorrect expectation, but a failing check should not turn green merely because its condition was relaxed. Inspect the code and the test changes together. If an assertion disappears, a configuration changes or a directory is excluded from analysis, require a justification tied to the acceptance contract. The number of passing tests matters less than the relationship between those tests and the specific risk introduced by the modification under review.

The identity of the tested version is also part of the evidence. Retain the repository revision, relevant configuration and results associated with that revision. If the patch changes afterwards, the earlier results do not automatically describe the new version. A clear report distinguishes checks that ran, checks that were skipped and checks that failed. Include the command and useful diagnostic information for reproducing the problem, without credentials, private data or sensitive screenshots. A test that could not run remains pending even if the rest of the work appears correct and ready to integrate.

Permissions, failures and gradual adoption

Code verification and tool authorisation solve different problems. A patch may pass its tests while an agent has uploaded private material to an inappropriate destination. Define which repositories, directories and services the task needs, then limit operations to that scope. If the task does not require creating public repositories, changing permissions or publishing screenshots, those capabilities should not be available by default. Explaining a restriction in the task helps communicate it, but does not replace a restriction enforced by the tool or execution environment itself.

When an operation fails, distinguish a known result from an uncertain one. An explicit test failure provides a diagnosis to investigate. A timeout while submitting a change can leave an operation completed without local confirmation. Before repeating it, inspect the destination state and look for the original revision or request. This habit avoids duplicate changes and preserves traceability. For every retry, record which condition has changed. Repeating the same action indefinitely with the same failure does not provide additional validation of the patch or evidence that the underlying problem has improved.

Choose the response to each class of failure in advance. If a style check fails, returning the diagnostic information to the agent may be reasonable. If expected behaviour changes, stop integration and return to the reproducible case. If the failure concerns access or publication, first check permissions and any effects already produced. Avoid granting more privileges merely to make a task finish. A stopped attempt with clear evidence is more useful than an apparently completed execution whose scope or consequences nobody can explain afterwards during review.

Before increasing autonomy, use bounded changes that can be reviewed realistically. Refactoring a function already covered by tests, for example, allows comparison between the previous and proposed behaviour without mixing in a new architecture. Keep the change small enough to inspect its consequences and reversal. Measure the work needed to accept or reject the proposal, including false alarms and manual corrections. The time spent generating a patch alone does not represent the total time required to maintain the repository, evaluate its output and resolve the exceptions that remain.

The final review should answer specific questions: what was requested, what changed, which evidence checks it and what remains uncovered. If repository policy allows automatic integration of this class of change, apply only that previously defined authorisation; otherwise retain responsible approval. Keep a way to return to the previous version and identify effects that reverting code cannot reverse. A useful barrier reduces observable risks and makes its limitations visible. A successful build, high coverage or an AI review does not become a universal guarantee of correctness or security.

Keep exploring

Corrections and transparency

If you spot incorrect or outdated information, send us a correction with the page and source we should review.