Open-Endedness Bench: What Can Be Claimed—and What Remains Unverified

The idea is relevant; the specific evidence is unconfirmed

Evaluating an agent by its final answer can be insufficient when it is working on an open-ended task. There may be no single solution against which to compare its result. In such cases, it also matters how the agent reached that result, what tests it performed, and whether it changed direction after encountering contradictory evidence. That is the need addressed by the proposal called Open-Endedness Bench (OEB), described in the editorial material provided for this review as a method for examining agents’ execution logs.

However, the documentation available for this review does not allow us to confirm the cited preprint, its contents, or the existence of accompanying materials. The original paper, evaluation instructions, execution logs, code, and reproducible results have not been provided either. It is therefore not possible to present as fact the criteria OEB supposedly applies, which agents it evaluated, or what scores they may have achieved. The proposal can be described as an idea awaiting verification, not as a validated benchmark.

The distinction matters: explaining why it could make sense to measure a research process is not the same as demonstrating that a particular method measures it reliably. In this article, hypotheses, experiments, and revisions are discussed as questions and useful criteria for assessing an evaluation of this kind—not as confirmed features of OEB.

What a research log would need to show

An evaluation of the process would need more than a sequence of well-written messages. To examine an agent’s operational reasoning, the log would have to distinguish between what the system proposed, what it actually did, and what results it obtained. Without that separation, a retrospective account could describe plausible experiments without showing that they took place.

It would also be necessary to reconstruct the connection between a hypothesis and a test. For example, if an agent proposes that a configuration change will improve a result, the log should show the configuration tested, the comparison conditions, the observed output, and the decision made afterward. It is not enough for all these pieces to appear somewhere in the record: they need to relate to the same question and retain enough context for someone else to review the interpretation.

Revising hypotheses is another potentially useful signal. An agent that retains an explanation despite incompatible results may be ignoring evidence; one that changes its conclusion should show which observation prompted the change. But a log alone does not automatically establish whether the revision was reasonable. That requires explicit criteria, a definition of what counts as relevant evidence, and—if human evaluators are involved—information about how their judgments were compared.

These are requirements worth looking for in any benchmark of automated research. They cannot be attributed to OEB unless a verifiable paper explains its protocol.

A final answer and a research process are different evaluation targets

Benchmarks for closed-ended tasks can compare an output with an expected answer or a verifiable result. In open-ended research, quality may not be reducible to getting one answer right. It can also depend on the experiments selected, the interpretation of incomplete data, and the ability to revise an explanation. This does not make final-answer metrics useless; it means they address a different question.

As limited context, OpenAI describes SWE-bench Verified as a human-validated subset of SWE-bench intended to evaluate models’ ability to resolve software issues more reliably. That example helps illustrate how task and outcome validation can form part of a benchmark’s design. It does not, however, show that a system evaluates open-ended research or measures the quality of an agent’s hypotheses.

To assess a proposal such as OEB, we would need to know at least which tasks it includes, which agents are compared, what units are scored, and how quality baselines are established. We would also need to know whether human judges validated the scores, whether agreement between them was measured, and whether the method was tested on misleading or incomplete logs—or on logs written without any actions actually being performed. Without those details, any eventual score cannot be interpreted as evidence of scientific rigor.

What is still needed to assess the proposal

The publication of a preprint would be only the first step in assessing the proposal. To check its methodological claims, we would need materials that make the evaluation reproducible: descriptions of the tasks, scoring rubrics, example logs, instructions for evaluators, and disaggregated results. Access to the code and original logs would also help establish whether scores depend on the format of the record or on particular decisions made by the research team.

Independent evaluation would need to examine both validity and robustness. Validity means that scores reflect the process they claim to measure, rather than, for example, the length of the log or the fluency of its prose. Robustness means checking whether conclusions hold across other agents, tasks, and evaluators. If the method uses human annotation, it would also be important to know the level of agreement and the cases of disagreement—not just an aggregate average.

OpenAI’s institutional page on SWE-bench Verified provides context about a separate benchmark and its human validation; it is not evidence that OEB underwent an equivalent evaluation. Likewise, a study of another agent benchmark does not confirm OEB’s data or design. Similarity in subject matter between projects is no substitute for a primary source for the study being discussed.

Conclusion: do not mistake a plausible proposal for an established result

The question raised by Open-Endedness Bench—how to evaluate what an agent does while conducting research, as well as what it answers at the end—is relevant to tasks without an obvious reference solution. But the verified sources provided do not confirm that the cited preprint exists under that title, or allow us to responsibly describe its methods or results. There is also not enough evidence to state which agents were evaluated, what scores they received, or whether their logs and tools are available.

The practical conclusion is provisional: before treating OEB as an operational benchmark, its primary source and materials need to be located and examined. Until then, the rigorous approach is to keep the general motivation—evaluating hypotheses, tests, and revisions—separate from specific claims about the method. That caution does not invalidate the idea; it avoids attributing results to it that have not yet been substantiated.

Keep exploring

Corrections and transparency

If you spot incorrect or outdated information, send us a correction with the page and source we should review.