In an Anthropic experiment, analysts followed their AI agent’s assigned lean in 86% of four decisions

What the experiment measured

Anthropic examined whether professionals working with an AI agent retain control over methodological decisions when more than one option is reasonable. The study focused on four analytical choices that could change the estimated effect of an agricultural program, followed by a final recommendation on whether to expand it.

The headline figure is a rate of agreement: across the four analytical decisions, the 50 participants who used the agent chose the option it had been instructed to favor 86% of the time. This does not mean that 86% of all experts always follow AI, or that the agent necessarily caused every matching choice. The publication reports the results of a specific task in which preferences were assigned experimentally.

How the study was conducted

The authors recruited 61 professional analysts with experience in observational causal inference and regular use of Python or R. Eleven chose not to use the agent and were excluded from the main analysis, which covered the other 50. The task presented a fictional case: deciding whether an agricultural cooperative should expand an advisory program for its members.

For each of four methodological decisions, a random assignment—a coin flip—determined which of two reasonable options the agent would favor. The choices included how to measure productivity, which comparison method to use, which farms to include in the analysis, and how to weight the results. The agent was instructed to make an honest case for its assigned preference, change its mind if the evidence was decisive, and not misrepresent the data.

The model’s preference was not left to emerge from its usual behavior: it was deliberately strengthened so the experiment could measure its effect. Analysts did not know which preference had been assigned. They worked in a web-based version of VS Code with Claude Code and Claude Opus 4.8; the task took a median of 65 minutes.

What the results showed

Agreement with the agent’s lean varied across the four choices: 92% for the outcome measure, 82% for the comparison method, 81% for the analysis population, and 90% for weighting. The combined figure was 86%. These percentages describe choices made by participants who used the agent in this task; they are not observations of professional AI use across different fields.

The difference in final recommendations was smaller: 54% recommended expanding the program when the agent leaned toward expansion, compared with 36% when it favored the alternative. However, the earlier methodological choices could change the estimated effect and therefore contribute to the recommendation. A statistical model using only the four assigned leans correctly predicted 82% of recommendations, compared with 61% when it always selected the most common recommendation. That result comes from the authors’ analysis and does not show that the same pattern occurs outside this task.

In 38 cases, analysts had stated a preference before the agent expressed its own. When the agent disagreed, analysts changed their choice in 60% of those cases; when it agreed, they changed it in 6%. The estimated difference was 54 percentage points, but it is based on a small number of cases and its 95% confidence interval was wide, from 24 to 82 percentage points. The authors also caution that analysts who stated a preference beforehand may have differed from those who did not, so this comparison alone does not establish a cause.

Adopted arguments and perceived authorship

After the task, 77% of analysts repeated the agent’s argument in at least one response about their decisions. In responses from 15% of participants, the agent was credited as a source at least once. Participants could not review their conversations when answering. These results show overlap with the arguments used by the agent, but they cannot by themselves establish why each person chose an option or whether they consciously remembered where an argument came from.

Perceived authorship did not necessarily reflect the observed agreement: 52% said the final recommendation was more their own than the agent’s. On average, those who considered it their own had also shifted toward the assigned leans. Confidence in the recommendation was similar among those who followed all four analytical leans and the rest: 71 versus 73 on a 0–100 scale.

The authors suggest one possible explanation: some people form an opinion while weighing the alternatives rather than starting with a firm preference. If an agent clearly argues for one option but says little about the other, it may occupy that decision-making space without the analyst noticing how much it contributed. The study documents a pattern consistent with that concern, but does not establish that it explains every choice observed.

What the study can and cannot establish

Randomly assigning the agent’s lean allows the authors to compare choices among participants exposed to different preferences and observe an average effect in the designed setting. Even so, the publication cannot establish how much the agent influenced each individual: analysts’ preferences were not fully recorded before the agent stated its position. Nor did the study measure how strongly analysts preferred each alternative.

The evidence comes from original work presented by its authors on Anthropic’s research page. The material consulted does not establish that the work underwent peer review or was independently replicated, so the figures should be understood as results reported by that study, not as scientific consensus. The available secondary source summarizes the experiment but does not independently verify its data.

There are clear limits to generalization: the study examined a fictional agricultural-analysis task, one model, and a group of causal-inference professionals who were not agricultural specialists. The authors also deliberately strengthened the agent’s preferences so they would be explicit and consistent. In everyday work, a model’s preferences may be less pronounced, decisions may differ, and experts may have greater subject-matter knowledge or stronger views of their own.

The study also compared those who used the agent with the 11 participants who did not. Six of the latter made serious methodological errors, compared with three of the 50 who used the agent; the median task time was 90 minutes for those who did not use it and 65 minutes for those who did. But the groups were not comparable: those who worked without AI were less experienced on average. The comparison does not show that using AI caused fewer errors or faster work.

Practical implications for working with agents

The practical issue is not limited to detecting calculation errors. Many professional tasks involve reasonable choices without a single correct answer: what to measure, which population to analyze, or how to weight results. An agent can produce a technically coherent response while steering decisions that depend on human judgment or values. Reviewing only the final result may not reveal those choices when either alternative is defensible.

One way to preserve informed human judgment is to ask a team to record its initial decision and reasons before receiving the agent’s recommendation, where feasible. The team can then compare that position with the agent’s arguments and ask it to make the strongest case for each alternative, including the assumptions and effects that change with each choice. These are process measures; the study did not evaluate them as interventions or show that they eliminate influence.

When a task contains many open decisions, requiring extensive review of every one can be costly and impractical. It may be more useful to identify in advance which choices could materially affect the conclusion, document the key decisions, and reserve time for an independent review of them. Organizations should also decide which aspects they can delegate and which require people’s own judgment, rather than assuming that a final sign-off automatically amounts to an independent review.

Keep exploring

Corrections and transparency

If you spot incorrect or outdated information, send us a correction with the page and source we should review.