Incident-Arena evaluates AI agents' incident response in simulated production environments

The gap in SRE agent evaluation

AI coding agents have become ubiquitous in engineering workflows, yet their ability to manage production incidents has received relatively little attention. The emerging field of agentic site-reliability-engineering (SRE) has relied on benchmarks with significant limitations.

These limitations include unrealistic environments, such as toy repositories, non-standard framework implementations, and simple static verifiers. Incident-Arena addresses these deficiencies by offering an environment that is more faithful to production conditions.

The reliance on toy repositories and non-standard framework implementations in previous evaluations created a significant disconnect between benchmark performance and real-world operational requirements. By moving away from these simplified static verifiers, the new approach aims to capture the nuanced complexity of production environments where incidents rarely follow predictable patterns. This shift is critical for understanding how agentic systems handle the unpredictability inherent in live infrastructure, rather than just solving isolated coding puzzles.

The emergence of agentic site-reliability-engineering highlights a specific gap in current assessment methodologies. While AI coding agents are increasingly integrated into daily engineering workflows, their capacity to manage the full lifecycle of a production incident remains underexplored. Addressing this deficiency requires an environment that faithfully replicates the pressure and systemic interdependencies of deployed systems, ensuring that the evaluation reflects the true operational burden placed on these autonomous agents.

Architecture of the 20 tasks

Incident-Arena is a human-built benchmark consisting of 20 carefully selected tasks. Each task is grounded in real-world open-source software that is deployed to an ephemeral Kubernetes cluster.

The fault injection process spans from the configuration layer through to the underlying images. Additionally, a specific sustained load profile is applied for each task, forcing agents to operate under pressure and consider the system-wide impact.

The human-built nature of the benchmark ensures that each of the 20 tasks is grounded in the complexities of real-world open-source software, avoiding the artificiality of synthetic scenarios. By deploying these applications to an ephemeral Kubernetes cluster, the environment provides a controlled yet realistic setting where the consequences of agent actions are tangible. This setup allows for the precise measurement of how agents interact with containerized infrastructure, a critical aspect of modern site-reliability-engineering that is often overlooked in simpler testing frameworks.

The fault injection process is designed to span multiple layers of the system, ranging from high-level configuration settings to the underlying container images. This multi-layered approach ensures that the challenges presented to the agents are not limited to a single point of failure but require a holistic understanding of the system architecture. Furthermore, the application of a specific sustained load profile for each task forces the agents to operate under continuous pressure, compelling them to consider the system-wide impact of their interventions rather than focusing solely on immediate symptom resolution.

Functional verification method

The benchmark introduces a novel verification method that goes beyond static checks. Instead of only reviewing code, it uses functional verifiers that evaluate the actual behavior of the application.

This approach keeps system-level metrics stable and ensures that repairs are made safely, avoiding regressions that could degrade the service. This allows for a more rigorous evaluation of the quality of the solution proposed by the agent.

Moving beyond static code reviews, the introduction of functional verifiers represents a significant methodological advancement in evaluating agent performance. By assessing the actual runtime behavior of the application, this method provides a more accurate reflection of the solution's effectiveness in a live environment. This approach ensures that the proposed repairs do not merely satisfy syntactic or structural requirements but genuinely restore the intended functionality of the service, thereby offering a more rigorous and meaningful assessment of the agent's problem-solving capabilities.

A crucial aspect of this verification method is the emphasis on maintaining system-level metric stability. By ensuring that repairs are made safely, the benchmark actively prevents the introduction of new issues or regressions that could degrade service quality. This safety-first evaluation criterion is essential for site-reliability-engineering, where the primary goal is not just to fix a bug but to do so without compromising the overall health and availability of the production system. Consequently, the quality of the agent's solution is judged not only on its correctness but also on its operational safety.

Results and test complexity

Agent trials in Incident-Arena require an average of 2.81 million tokens and 41 interaction turns. This magnitude exceeds that of existing benchmarks, demonstrating the need for long-horizon reasoning by agents.

Across the 20 tasks and 3 application substrates, frontier models score below 64.3%. Observed failures include errors in diagnosis and localization, incomplete repairs, and unsafe regressions.

The computational demands of the agent trials in this environment are substantial, with an average requirement of 2.81 million tokens and 41 interaction turns per task. This magnitude significantly exceeds the resource consumption observed in existing benchmarks, highlighting the necessity for long-horizon reasoning capabilities in autonomous agents. The extended interaction sequences indicate that resolving these incidents is a complex, multi-step process that requires sustained attention and strategic planning, rather than a simple, one-shot code generation task.

Despite the advanced capabilities of frontier models, their performance in this rigorous setting remains limited, with scores falling below 64.3% across the 20 tasks and three application substrates. The observed failure modes provide valuable insights into current limitations, including errors in initial diagnosis and fault localization, as well as the provision of incomplete repairs. Additionally, the occurrence of unsafe regressions underscores the difficulty agents face in balancing the need for a fix with the requirement to maintain system stability, revealing a significant gap between current model capabilities and the demands of production-grade site-reliability-engineering.

Availability and study limitations

The work is presented as a preprint on arXiv, meaning it has not yet undergone a formal peer-review process. The authors acknowledge that the evidence provided in the abstract is not sufficient to verify all technical details, such as exact software versions or full reproducibility.

Although the benchmark offers a more realistic view of SRE agent capabilities, its results must be interpreted with caution. Operational reliability in real production environments cannot be directly extrapolated based solely on these experiments.

As a preprint, the work has not yet undergone the rigorous scrutiny of a formal peer-review process, which means that certain technical details remain unverified by the broader scientific community. The authors themselves acknowledge that the evidence provided in the abstract is insufficient to confirm all aspects of the study, such as the exact software versions used or the full reproducibility of the experimental setup. This status necessitates a cautious interpretation of the findings, as the lack of independent validation means that some methodological details may still be subject to refinement or correction.

While the benchmark offers a more realistic and comprehensive view of SRE agent capabilities compared to previous efforts, its results must be interpreted with appropriate caution. The operational reliability observed in these controlled experiments cannot be directly extrapolated to real production environments, which are characterized by greater variability and unpredictability. Therefore, while the study provides a valuable foundation for understanding agent performance in complex scenarios, it should be viewed as a preliminary step in a larger effort to establish robust standards for evaluating autonomous site-reliability-engineering tools.

Keep exploring

Corrections and transparency

If you spot incorrect or outdated information, send us a correction with the page and source we should review.