An audit separates why AI agents fail when searching scientific literature

The limits of evaluating only the final answer

Traditional evaluations of scientific-search agents focus on the accuracy of the final answer. But a single accuracy score combines different kinds of failure: an agent may not find the target paper, may find it without inspecting it, or may inspect it and still produce an incorrect answer.

That lack of visibility makes systems harder to improve because it does not identify where the error occurred. A preprint by Hongmin Li and Wanli Zhao proposes an audit method that records intermediate events during inference, allowing failures to be classified in more detail. The central idea is to examine not just what an agent answered, but also what it encountered and what it attempted to do along the way.

Decision checkpoints during the search

The study introduces “decision checkpoints” that record observations and tool actions during inference, without using benchmark labels while the agent is running. The records are used to determine whether the agent was exposed to the target paper and whether it attempted to inspect it.

After inference, the target identities and evaluator labels are joined to the recorded events to assign outcome categories. This makes differences visible question by question and separates target exposure and inspection from both aggregate accuracy and total tool use. The method is designed to classify events in the search process rather than to rely on the final answer as the sole measure of what happened.

Experimental conditions and reported comparisons

The experiment covered 540 answerable questions from AutoResearchBench Deep, across five conditions, in a fixed environment enriched with target documents. That setting is important to interpreting the results: the reported numbers describe these questions and this experimental configuration, not scientific search in general.

Within this setup, keyword search achieved 24.6% accuracy, compared with 17.8% for raw search. The study also reports that read-first recorded 27.4% more evidence-search calls than keyword search. These figures describe different aspects of performance: accuracy measures accepted answers, while evidence-search calls count recorded tool activity.

Separating retrieval, inspection, and answer failures

The event categories show why a single accuracy score can conceal different patterns. Compared with raw search, keyword search had fewer incorrect answers in which the target had been neither exposed nor inspected. At the same time, keyword search had more incorrect answers in which the target had been exposed but left uninspected. Those categories distinguish failing to encounter the paper from encountering it without attempting to inspect it.

Recorded attempts to inspect the target occurred on 199 questions under read-first and 191 under keyword search. Both conditions achieved 24.6% accuracy. In this experiment, therefore, more recorded inspection attempts did not correspond to higher aggregate accuracy. The comparison does not establish why the answers were wrong in individual cases, nor does it show that inspection is generally ineffective; it shows that inspection attempts and accepted answers are separate measurements.

Why these results do not establish general reliability

The findings come from a fixed environment enriched with target documents. That configuration may differ from open-ended scientific search, where relevant papers are not deliberately supplied as targets and the available material can be broader or less predictable. The preprint’s abstract does not provide enough detail to specify exactly how the enriched environment was constructed or to quantify the biases it may introduce.

The work is a preprint, submitted on September 29, 2026. The available source establishes that it was posted as an arXiv preprint; it does not establish peer review or independent replication. Its reported results should therefore be read as findings from one study and one stated experimental setup, not as evidence that AI search agents are reliable—or unreliable—in every scientific-search context.

Keep exploring

Corrections and transparency

If you spot incorrect or outdated information, send us a correction with the page and source we should review.