NP-Bench tests coordination between parallel coding agents

The problem does not end when each agent passes its tests

Several coding agents can work simultaneously in the same repository and still produce a result that does not integrate correctly. Each agent may complete its task and pass its individual tests, while the combined result fails because changes overwrite one another or one agent relies on an interface another has just changed.

Sumanyu Muku’s preprint argues that evaluating agents separately cannot reveal this kind of collective failure. Its proposal, called NP-Bench, treats coordination as something that can be tested by integrating actual changes, rather than evaluating only the quality of each contribution in isolation. The arXiv record identifies the work as a preprint submitted on October 5, 2026; the available evidence does not establish that it has undergone peer review.

The proposal: plan scope and integration order

The planner described in the abstract organizes work before tasks are carried out. First, it takes the declared scope of each work item and divides the work into disjoint areas. It then orders integrations according to dependencies between producers and consumers: in practical terms, it aims to integrate a change that establishes a condition before incorporating another change that depends on it.

This idea contrasts with a reactive workflow that waits for a collision to be detected before issuing a warning. Anticipating dependencies could prevent wasted work when a warning arrives after agents have already developed against incompatible assumptions. For now, however, that advantage is the hypothesis the paper seeks to evaluate: the abstract does not provide enough detail to assess how scopes are declared, how ambiguous dependencies are resolved, or how costly it is to keep the plan up to date.

What NP-Bench compares and what results it reports

The abstract describes NP-Bench as a benchmark with three conditions: work without coordination, reactive conflict detection, and proactive planning. It also says that integration is checked using an actual Git merge and that the benchmark includes both a deterministic simulation and trials with live agents. As described, its main metric is therefore not simply whether each task is completed, but whether the combined work integrates cleanly.

In the simulation, the abstract reports that clean integration rose from 1 of 9 scenarios to 9 of 9 with the planner, while merge conflicts fell from 13 to 0. For a live test involving a breaking contract change, it reports a clean-integration rate of 1.0 with a frontier model and 0.6 with a small model, compared with 0 for both comparison conditions. These are results attributed to the preprint’s own experiment, not a guarantee of performance in other repositories or teams.

What evidence is needed to assess the improvement

To understand how strong these results are, the full paper needs to be examined to establish how the tasks were constructed, which repositories or environments were used, and exactly what counted as clean integration. It also matters how scopes were assigned, how simultaneous changes were handled, and how many runs support each rate. The available bibliographic record provides the abstract, not all those details of the protocol.

The comparison should make it possible to distinguish the planner’s effect from other factors, including model capability, instructions, task structure, and merge policy. It would be useful to inspect results broken down by scenario and number of agents, as well as failures and exceptions. The abstract says that the benefit did not shrink with more capable models and that two capability tiers and two vendors were tested; without configurations and disaggregated results, it is not possible to assess the breadth of that comparison independently.

Limitations: a preprint and results that still need context

The evidence provided comes from the preprint’s arXiv page and its abstract. The numerical results in this article should therefore be read as results reported by the author, not as independently replicated findings. The available documentation also provides no basis for describing the work as peer reviewed or claiming that the method has already been shown to improve integration in real development teams.

The abstract mentions other findings: agents respected their assigned scopes in the case described; cross-session memory reduced repeated mistakes; and routing facts to agents did not resolve accuracy issues in long contexts when the content fitted within the context window. These are claims from the same work and fall outside the main evaluation of planning unless the full methods and results are examined. In particular, a favorable result should not be turned into a general promise about every team, codebase, or model.

Provisional verdict: a research question, not a proven solution

NP-Bench raises a useful question for evaluating coding agents: does integration improve when task scopes and ordering are planned before changes come into conflict? The contribution described is to measure that difference against both uncoordinated work and a reactive response to problems, checking integration through a Git merge.

The abstract reports favorable results for the planner in the evaluated scenarios, but the available material does not make it possible to assess the full methodology or claim that the effect holds under production conditions. The responsible conclusion is provisional: this is a research proposal with initial results reported by its author. Its practical relevance will depend on whether the full paper and materials make it possible to verify the comparison, and whether later evaluations confirm its usefulness across diverse projects.

Keep exploring

Corrections and transparency

If you spot incorrect or outdated information, send us a correction with the page and source we should review.