AlphaIdeaBench evaluates research ideas against budgets and a 12-dimension rubric

An evaluation that accounts for available resources

Generating a research idea is not just about proposing something novel. It also matters whether the plan can be carried out with the available time, computing power, and API credit. AlphaIdeaBench addresses this tension with a benchmark designed to compare automatically generated research proposals under explicit budgets, rather than scoring only their novelty or similarity to published work.

The paper was submitted to arXiv on October 24, 2026, and is presented as a preprint. Its abstract describes the design and some results, but does not by itself establish that the conclusions have undergone peer review or been reproduced by other teams. AlphaIdeaBench is therefore best understood as a proposed measurement framework and an initial set of findings, not as a general validation of research agents.

One hundred tasks with explicit conditions

AlphaIdeaBench contains 100 tasks: 50 come from workshop calls for papers and 50 from open problems described in survey papers. Each task is paired with a resource budget and a shared proposal format. This structure is intended to make responses comparable against common criteria and within stated limits.

Including a budget changes the evaluation question. A proposal may seem interesting in the abstract while requiring more computing power, time, or API credit than is available. The benchmark makes that distinction part of its analysis, although feasibility scores remain evaluations by the system, not proof that a plan has actually been executed. The available abstract does not give the values of every budget or allow readers to inspect each individual task.

Twelve dimensions for assessing proposals

The AI judge assigns scores across 12 dimensions grouped around validity, novelty, and feasibility. The paper distinguishes four axes of novelty: novelty of the problem, the method, the scenario, and the insight or knowledge contributed. Separating these axes can help prevent a single overall judgment of “originality” from obscuring important differences—for example, between a new research question and a different technique.

Budget feasibility also matters: a proposal that exceeds the assigned resources does not meet the same goal as one that fits within them. Even so, a rubric applied by a model is neither a deterministic control nor proof that a plan can be carried out. Scores depend on how the criteria are defined and how the judge interprets the proposals; the abstract does not provide enough detail to reconstruct all the scoring rules.

The judge is compared with reviewer comments

To examine the judge, the study compares its scores on four novelty axes with scores derived from reviewer comments on 600 papers from ICLR 2026, NeurIPS 2025, and ICML 2025. The abstract reports that Gemini 3.7 Flash came within one point of the review-derived score for 94–97% of papers on problem and method novelty. Agreement was relatively lower for scenario novelty.

That percentage describes closeness to a score derived from review comments on the specified axes; it does not mean the judge and reviewers agree on every criterion, nor that the model correctly evaluates any new proposal. The abstract does not say how many reviewers contributed to each paper, how their opinions were aggregated, what exact measure defines “within one point,” or how much reviewers agreed with one another. The comparison therefore offers a limited signal of alignment, not a general certification of quality.

Differences between agents and backbone models

The paper compares eight published systems, called harnesses, using a shared backbone, and also compares eight language models within one of those harnesses. According to the abstract, CoI-Agent has the highest overall score, narrowly ahead of SciMON; SciMON leads on method and insight novelty. All eight systems score low on scenario novelty and rarely change the problem setting.

The authors report that the backbone model largely determines which open problem a proposal targets, while the harness influences whether a response addresses one problem or combines several. The abstract also says that weaker models, such as Llama 4 Scout, struggle to follow the evaluated harness’s strategy and score low on budget feasibility. These are results from the specific set and configuration examined; they do not establish a lasting ranking of models or necessarily predict their behavior on other tasks.

A larger budget can change experimental design

In an additional analysis, the researchers varied only the budget across three topics. For compute-intensive topics, a larger budget changed the experimental setup. For the inexpensive topic, it added tests instead, with less change to the overall design. This observation illustrates why the same increase in resources does not necessarily have the same effect on different problems.

Because this test covers three topics, it cannot establish that the pattern holds across all scientific fields. It does point to a practical distinction when reading proposals: more resources may allow a different kind of experiment when computing costs are a major constraint, while in other cases they may mainly enable additional checks. In either case, a proposal needs independent assessment before it can be treated as an executable plan.

Scope and limits of the available evidence

AlphaIdeaBench proposes a concrete framework for comparing ideation agents: budgeted tasks, a shared format, a multidimensional rubric, and a partial comparison with review comments. Its potential value lies in making evaluation conditions more explicit and enabling comparisons within a defined set, rather than reducing assessment to a general impression of novelty.

The arXiv record consulted identifies the paper as submitted in October 2026; the material available for this article is the abstract, not an independent verification of every methodological detail. The abstract says that tasks, rubrics, and scores will be released, but does not by itself confirm that those materials are accessible or allow readers to check code, data, or reproducibility. It also does not report the number of reviewers or all the details needed to assess how human scores were constructed. These limits constrain what can be said about the strength and generalizability of the results.

Keep exploring

Corrections and transparency

If you spot incorrect or outdated information, send us a correction with the page and source we should review.