InfraDev evaluates coding agents on 36 AI infrastructure changes

A benchmark of infrastructure changes, not just code snippets

InfraDev is a benchmark designed to assess whether coding agents can implement complex changes to artificial intelligence infrastructure. It focuses on tasks that span multiple files and must run in multi-GPU environments, rather than isolated kernels or single-GPU jobs.

That distinction matters because a change can appear correct in a small function yet fail when integrated into a training or inference system. Such systems involve interfaces, shared state, dependencies, and computational resources. InfraDev attempts to capture some of this complexity with tasks based on real changes. Its results, however, describe performance on this particular task set—not an agent’s general ability to maintain any production system.

36 tasks drawn from projects used to develop models

The benchmark contains 36 tasks built from pull requests to Megatron-LM, vLLM, and verl. They cover model training, inference, and reinforcement learning. In each task, the goal is to reimplement a real change rather than complete a standalone programming exercise.

This connection to existing infrastructure software gives the tasks a practical grounding, but it does not make the 36 examples a comprehensive sample of changes that could arise in those projects. The available abstract does not establish that every type of work, architecture, or hardware configuration is represented. The projects and areas covered therefore provide context for interpreting the evaluation, not evidence of universal coverage.

Hidden probes look for failures that official tests may miss

InfraDev combines hidden tests, called probes, with an instruction that specifies the interfaces those probes use without revealing the solution. The aim is to check requirements that may be absent from a project’s original tests and reduce the chance that an answer appears valid simply because it passes an incomplete set of checks.

The paper’s abstract reports an important gap: the probes rejected 32% of submissions that had passed every official test. For the tasks evaluated, this indicates that passing the official suite was not always enough to satisfy the benchmark’s additional requirements. It does not show that hidden tests are infallible or that this method can fully assess every real-world change.

The best reported result solved 38.9% of the tasks

The evaluation tested six frontier models, and the best result solved 38.9% of the tasks. When the original pull request descriptions were used as the prompts, no model solved a task. This contrast suggests that how interfaces and requirements are presented can affect what agents accomplish in this benchmark.

The 38.9% figure is not a success rate for all infrastructure engineering work. It applies to InfraDev’s tasks and evaluation conditions as described in the preprint. The abstract does not provide enough detail here to independently compare every model, task, and project, so the top result should not be treated as a general ranking of coding agents.

Structural and state-related errors were common

The paper’s failure analysis reports that structural errors and errors involving state were prevalent. It also notes that agents’ own tests rarely checked factors such as memory, execution time, or traffic. These are observations about the tasks and responses analyzed, not proof that the same failures characterize every agent or project.

The paper also reports that turning its findings into an agent skill raised a frontier model’s resolution rate by 10.5 percentage points on the Megatron-LM tasks. That result is specific to the model and subset described; it does not establish that the improvement would carry over to other projects or systems. The authors also describe a shared execution environment that reduced storage by 52–57% and build time by 13–14% on Megatron-LM and vLLM.

A preprint offering a concrete test, not universal validation

InfraDev is presented in a paper hosted on arXiv as version 1, submitted on October 27, 2026. The available source is the paper’s own abstract; it does not establish that the work has undergone peer review or been independently replicated. The reported figures should therefore be understood as results communicated by the authors in this preprint.

The benchmark offers a test closer to real infrastructure tasks than one limited to kernels or a single GPU, but the available evidence does not settle questions about representativeness, coverage, or performance in other deployments. Its most defensible conclusion is limited: under the tasks and conditions described, the evaluated models struggled, and additional tests detected failures that the official tests did not catch.

Keep exploring

Corrections and transparency

If you spot incorrect or outdated information, send us a correction with the page and source we should review.