A preprint proposes checking whether AI agents deserve their benchmark scores

A score does not tell the whole story

An agent can receive a passing score on a benchmark. That result confirms that the system received credit under the evaluation procedure used, but it does not necessarily explain what the agent did to earn it, whether its claim to have completed the task is true, or whether it would get the same result on another run. This gap between receiving credit and verifiably demonstrating capability is the problem addressed by Trust Layer, a framework presented in a preprint.

The proposal does not replace or modify the score. Instead, it adds checks after evaluation and reports whether the result passes four trust checks. The aim is to separate two questions that are often combined in a single number: what score the agent received, and what evidence supports the belief that the score reflects genuine, reproducible performance.

The paper applies the framework to five agent configurations across 108 tasks from Agents’ Last Exam. Its figures therefore describe that specific experiment; they should not be read as universal estimates of the reliability of all agents, models, or benchmarks. Nor do they show that the framework has been adopted as a standard or independently validated.

Four different questions about a passing result

The first check asks whether the result is supported by the benchmark’s own grading logic. Looking only at the final score is not enough: the recorded answer must be examined to see whether it meets the conditions the evaluator was supposed to apply. This check concerns the relationship between the evidence and the scoring criteria; it does not establish whether those criteria, by themselves, measure a useful real-world capability.

The second check asks whether a passing answer was produced through a traceable computational process. Its purpose is to distinguish a result accompanied by verifiable steps or evidence from one that does not leave enough information to show how it was reached. “Not traceable” does not automatically mean “incorrect”: it means that, based on the artifacts examined, the required process cannot be verified.

The third check compares the agent’s completion claim with what actually happened. For example, a system might declare that it finished a task even though the saved evidence does not show that the action was completed. The preprint’s summary reports confirmed false completion claims, but does not give a specific rate for this category. It would therefore be inappropriate to assign it a frequency.

The fourth check asks whether the result holds up when the agent is run again. A single run cannot measure this kind of stability. In the experiment described, the check involves repeating the run and comparing the resulting scores. A one-off pass and consistent performance are therefore different signals.

What can be checked in saved records—and what requires another run

According to the preprint’s description, the first three properties are assessed using saved artifacts. In practical terms, this means the audit depends on relevant records being available—for example, answers, evaluator results, and evidence of actions—and on those records being linkable to the task. The summary does not specify which exact formats are required for each check or which artifacts were missing in the cases studied.

Stability requires something different: the agent must be run again. The paper measured it over five runs and reports that between 18% and 46% of tasks, depending on the configuration evaluated, did not remain in the same score band. The range indicates variation across the configurations studied; it does not mean that the same proportion applies to every agent or benchmark.

The framework also combines model judgments with deterministic rules. Model judgments are used to label evidence under majority voting, while the final verdicts follow deterministic rules. This distinction matters: the method does not present auxiliary model judgments as conclusive proof in themselves, and it states that the checks do not alter the recorded score.

Notable results, but limited to this experiment

Across the five configurations analyzed, the summary reports passing runs without traceable computation. The frequency varied by as much as a factor of ten between models. This points to substantial differences within the set studied, but does not by itself explain why they occurred or establish which configuration would be more reliable on different tasks.

The paper also found confirmed cases of false completion claims and variation between runs. Overall, just 22.6% of recorded passes cleared all four checks. The reported 95% confidence interval is 15.0% to 32.6%, with n=84. This figure applies to the passes included in that calculation, not to every possible run or to a general population of benchmarks.

The appropriate interpretation is not that agents can correctly perform only some fixed fraction of tasks. The result describes the share of passing results in this experiment that simultaneously cleared these four checks. If a check fails, the conclusion depends on which one: there may be doubts about the evaluator, a lack of traceability, an unsupported success claim, or instability across repeated runs.

How it differs from other ways of evaluating agents

Agent tests commonly examine aspects such as planning, actions, tool use, and system responses. Trust Layer does not replace that kind of evaluation: it proposes adding a review of the evidence behind the credit awarded. Rather than asking only whether the agent reached the expected answer, it also asks whether the result can be verified and whether it remains stable when repeated.

There is a related but distinct problem: whether a benchmark genuinely represents the capability it claims to measure. An independent study on auditing agent benchmarks analyzes traces across 15 benchmarks and examines when their scores support claims about capability. That question—benchmark validity—is not the same as checking whether a particular result was traceable or whether a completion claim matched what happened. The two lines of work may complement each other, but neither automatically establishes the other.

The available information does not support a claim that Trust Layer outperforms existing audit methods or improves benchmark quality. The summary describes the checks and the results of one application, but does not establish a systematic comparison with other methods. Assessing whether it offers an advantage would require protocols, criteria, and results to be compared under equivalent conditions.

What remains unknown about its usefulness beyond the study

The proposal offers a structured way to accompany a score with auditable questions, but the available evidence does not confirm whether the code, artifacts, or data needed to reproduce the checks have been published. Nor does it establish in detail what operational decisions each rule requires. Without those materials, it is difficult for other teams to check whether they would reach the same verdicts.

The evaluation described is limited to five configurations, 108 tasks from one benchmark, and five runs for measuring stability. Applying the framework to other task types, evaluators, and agents would help establish whether the properties hold and how differences in record quality affect the results. It would also be important to study what happens when artifacts are missing or when the scoring logic does not permit an unambiguous verification.

For now, Trust Layer should be understood as a methodological proposal supported by the results of one specific application, not as an established standard or a guarantee that a score is true. Its central contribution is to ask that benchmark credit be accompanied by evidence about how it was awarded, what happened during the task, and whether the result holds up when repeated. The usefulness of that approach will depend on whether it can be reproduced and tested in more settings.

Keep exploring

Corrections and transparency

If you spot incorrect or outdated information, send us a correction with the page and source we should review.