Anthropic reports defensive cybersecurity gains for Claude Sonnet 4.5, with important limits
Image generated by the Inferama Editorial Team
01

Anthropic targets Sonnet 4.5 at defensive tasks

Anthropic describes research intended to improve Claude’s usefulness for defensive cybersecurity. The company focused part of Claude Sonnet 4.5’s development on finding vulnerabilities in code, proposing fixes and testing weaknesses in simulated infrastructure. Its stated goal is to help identify flaws before deployment and remediate them in existing systems.

Anthropic says Sonnet 4.5 matches or exceeds Claude Opus 4.1 in several cybersecurity tasks and is also faster and less expensive. That comparison describes performance the company observed in its tests; it does not mean the model outperforms security professionals at every task or is ready to replace established processes.

Anthropic says it prioritized defensive tasks, such as finding and fixing vulnerable code, and avoided improvements clearly directed toward offensive uses, such as advanced exploitation or writing malware. That choice defines the research’s intended purpose, but does not remove the dual-use nature of cybersecurity capabilities: finding a vulnerability can help fix it and can also make exploitation easier.

The publication presents the results as part of a strategy to accelerate defensive AI adoption. It also acknowledges that the evaluations do not cover every security task and that real-world problems usually involve greater complexity and more implementation details than laboratory exercises.

02

What the evaluations measured

Anthropic evaluated Sonnet 4.5 using Cybench and CyberGym, two tests that cover different tasks. Cybench draws on Capture-the-Flag competition challenges. CyberGym tests whether an agent can reproduce known vulnerabilities in open-source projects and discover new ones. The results do not measure general cybersecurity ability; they reflect performance on these task sets under specific conditions.

Anthropic emphasizes that it repeated tasks multiple times. In a probabilistic test, more attempts can raise the chance that at least one succeeds. A success rate based on ten or thirty attempts is therefore not equivalent to the result of a single run, or to the probability of success on any real-world system. Anthropic says repeated attempts are intended to approximate how a motivated attacker or defender might approach a particular problem.

On Cybench, Sonnet 4.5 solved 76.5% of challenges with ten attempts per task, Anthropic reports. The figure covers a subset of 37 of the original 40 problems; three were excluded because of implementation difficulties. The company also says Claude took 38 minutes on a task involving network-traffic analysis, malware extraction and analysis of that program. That duration applies to one challenge, not to a general average.

On CyberGym, Anthropic reports a score of 28.9% with a two-dollar limit on API queries per vulnerability, the same cost constraint it attributes to the public leaderboard. With thirty attempts per task and without that limit, the company says the model reproduced vulnerabilities in 66.7% of programs. The second figure came from a different condition, at an estimated cost of about 45 dollars per task; the two measurements should not be treated as equivalent.

For discovering new vulnerabilities in CyberGym, Anthropic reports a 5% success rate with one attempt and more than 33% of projects when repeating tasks thirty times. These figures describe the specified evaluation. They do not establish how many new vulnerabilities the model will find in production or how long it will take.

How to interpret the main reported figures

The metrics depend on the task, number of attempts and cost limit.

TestConditionReported resultScope
CybenchTen attempts per task76.5% of challenges solvedSubset of 37 problems; the metric counts success if at least one attempt works.
CyberGym: known vulnerabilitiesTwo-dollar limit per vulnerability28.9%Cost condition used for the comparison described by Anthropic.
CyberGym: known vulnerabilitiesThirty attempts per task, without the same cost limit66.7% of programsEstimated cost of about 45 dollars per task; not directly comparable with the previous row.
CyberGym: new vulnerabilitiesOne attempt / thirty attempts5% / more than 33%Discovery results in this evaluation, not a universal rate for real systems.
03

Results depend on attempts and test conditions

Cybench and CyberGym make it possible to compare results in defined scenarios, but they do not reproduce the full range of networks, applications, permissions and processes in an organization. Anthropic warns that real-world problems are more complex and that implementation details matter. A benchmark score therefore does not show that a model can operate safely in a particular infrastructure.

The number of attempts changes what an evaluation measures. If each attempt provides another chance to solve a task, the success rate across a set can rise even if the chance of getting it right on the first run does not change. When reporting results, the percentage, benchmark and number of attempts should remain together.

The cost limit matters too. CyberGym’s score under the two-dollar limit answers a different question from the thirty-attempt test, which Anthropic estimates cost about 45 dollars per task. The latter approximates a scenario with more resources, but it does not show that the additional performance is inexpensive at scale: costs may accumulate as testing expands.

Anthropic says Sonnet 4.5 outperformed earlier models in the evaluations described. Because the results are reported by Anthropic and the publication does not provide independent validation of every result, the figures should be attributed to the company. Using external evaluations developed by third parties can reduce the risk of tuning a model solely to internal tests, but does not amount to independent replication.

04

Patching remains preliminary

Anthropic also tested whether Sonnet 4.5 could generate patches for CyberGym vulnerabilities. The model received a description of the flaw and information about what happened when the program crashed. Claude then evaluated the patches by comparing them with human-written reference solutions. The company reports that 15% were judged semantically equivalent to the references.

That figure is not a demonstrated rate of valid fixes in production. The method can mark a functional patch as incorrect if it fixes the problem in a different way, because the comparison uses a particular reference solution. Anthropic explicitly acknowledges this limitation and describes the work as preliminary.

Anthropic manually reviewed a sample of the highest-scoring patches and says some were functionally identical to reference patches merged into the open-source software being evaluated. Reviewing a sample does not justify generalizing the result to every generated patch. A patch needs testing and review to confirm that it fixes the vulnerability without changing intended functionality.

Anthropic considers patch generation harder than vulnerability discovery: the model must make limited changes while preserving expected behavior, which may not be explicitly specified. The publication describes systematically addressing these problems as a next step toward a model that can write and review patches reliably; it does not claim that this goal has already been achieved.

Carefully evaluating an AI-proposed patch

The process retains human review and testing in the system where the fix would be applied.

  1. 01Reproduce the flaw in an authorized environment and record the test conditions.
  2. 02Inspect the proposed change and confirm that it addresses the cause of the vulnerability.
  3. 03Run functional and security tests to check for regressions.
  4. 04Obtain review from a responsible person before integrating the patch.
  5. 05Document the outcome and keep a rollback option if the fix causes problems.
05

Defensive applications and deployment limits

Anthropic says it worked with organizations that tested Sonnet 4.5 on tasks such as vulnerability remediation, network-security testing and threat analysis. The publication includes statements from HackerOne and CrowdStrike about possible applications of the model. Those comments describe the experiences of the people quoted; they are not a controlled evaluation with detailed results for every organization.

Anthropic identifies security operations center automation, security information and event analysis, secure network engineering and active defense as areas for further exploration. These are areas the company wants to explore, not capabilities whose effectiveness in all those settings has been established by the Cybench or CyberGym figures.

The publication also mentions that Claude Code can integrate automated security reviews into a continuous integration and deployment workflow. Adding a review does not make every alert a confirmed finding or every proposed fix a safe change. Teams still need to verify results, limit permissions and apply changes in suitable environments.

Anthropic says its teams use techniques such as organization-level summaries to interpret activity beyond a single interaction and better distinguish dual-use behavior from malicious activity. The publication does not provide metrics to quantify how much these measures reduce misuse. They should therefore be understood as described safeguards, not as a guarantee that misuse will be prevented.

The company acknowledges that some of Sonnet 4.5’s capabilities remain nascent and do not yet match security professionals or established processes. For an organization, the practical choice is not to replace those processes, but to test narrowly defined tasks with controlled permissions, verifiable outcomes and human review.

What an organization can conclude from these results

Local evaluation should distinguish tested tasks from research goals.

Proposed useWhat the publication supportsWhat the team should check
Find vulnerabilities in codeAnthropic’s results on Cybench and CyberGymAccuracy in its repositories, false positives, coverage and permissions used.
Propose a patchPreliminary experiment comparing patches with human referencesFunctional correctness, regressions, security and review before integration.
Analyze networks or threatsApplication areas and partner statements mentioned by AnthropicPerformance in its environment, access limits and validation of each finding.
Automate security operationsResearch direction Anthropic wants to exploreWhich tasks can be automated without delegating critical decisions to the model.
06

What changes—and what has not been demonstrated

Anthropic’s main change is to dedicate research to defensive cybersecurity skills rather than relying only on general model improvements. The company reports favorable Sonnet 4.5 results on specific vulnerability-detection and reproduction tests, while presenting patch generation as promising but still preliminary.

The figures help explain performance under explicit conditions: benchmark, number of attempts, cost and task subset. They do not prove that the model finds every flaw, always produces correct patches or is safe in every deployment. Defensive usefulness depends on context and verification.

The publication also makes clear that cybersecurity covers more than the evaluated tests. Securing deployed systems requires understanding configurations, permissions, dependencies and operational consequences. Anthropic does not claim that Sonnet 4.5 replaces human teams; it describes experiments with organizations and acknowledges that reliable patch generation and review still require further work.

Teams evaluating a tool of this kind can start with a limited scope: choose authorized repositories or systems, define which operations the model may perform, measure correct findings and false positives, review each change and compare results with the existing process. This approach does not remove dual-use risks, but it lets an organization test local usefulness without mistaking a published score for a guarantee.

07

Keep exploring

Sources consulted

03

Corrections and transparency

If you spot incorrect or outdated information, send us a correction with the page and source we should review.

Submit a correction