OpenProblemBench evaluates AI on 82 unresolved scientific problems

An evaluation beyond questions with known answers

OpenProblemBench is a benchmark designed to test AI systems on unresolved problems in mathematics and theoretical physics. Published as a preprint on arXiv on October 8, 2026, the work presents 82 problems drawn from the literature in those fields. The proposal aims to assess something different from answering questions with established solutions: whether a model can make valid contributions toward a scientific question that remains unresolved.

An answer can sound convincing without constituting mathematical or scientific progress. For open problems, there is not always a reference solution against which an answer can be checked automatically. The benchmark attempts to address that challenge by providing research context and using evaluators to judge submissions. The paper’s abstract describes its design and results, but on its own it is not enough to establish how every criterion was applied to every problem.

What the problem set contains

The authors describe the 82 cases as unresolved problems drawn from the mathematics and theoretical physics literature. For each one, the benchmark supplies research context, assumptions and prior progress. This is intended to give systems more than an isolated question: they receive information relevant to investigating the problem.

The selection focuses on questions whose decisive mathematical or computational claims can be checked comparatively clearly. That choice defines the kinds of problems included: the set does not necessarily represent every area of theoretical science or every challenge in open-ended research. The available abstract does not give case-by-case inclusion criteria, explain how the authors established that each problem remained open, or specify what documentation accompanies each problem statement. Those details need to be checked in the full paper and associated materials before drawing conclusions about coverage or reproducibility.

How submissions are judged

According to the preprint’s abstract, four evaluator models independently assess each submission. They rate three dimensions: correctness, completeness and degree of progress. Evaluation proceeds without reference solutions. That choice makes it possible to work with open problems, but it also makes the rubric and the limits of automated judgment especially important.

The abstract does not say that the four evaluators are human specialists or formal verifiers, nor does it describe a comparison against an expert standard. Their judgments should therefore be understood as model-generated evaluations, not independent certification of a mathematical proof. Checking a decisive computational claim may be more straightforward in some cases; validating a proof or establishing that an argument covers every case may require specialist review. The information provided does not specify which verification methods were used for each task.

Results reported in the preprint

Across seven evaluated configurations, the paper reports that GPT-6-Astra achieved the highest mean judged solve rate, at 14.0%. The evaluated full-size open models scored between 5.5% and 6.7%, while the Flash configurations scored between 2.4% and 3.7%. These figures describe the reported results for the configurations studied; they are not a universal measure of scientific capability or a comparison of every available model.

The authors also link stronger results in case comparisons to changes in how problems are represented, general arguments that go beyond finite evidence, and proofs of the steps needed to complete a solution. These observations offer clues about the answers that received better evaluations, but the abstract does not support the conclusion that any one pattern causes general improvement or will carry over to other disciplines or benchmarks. Nor does a positive score mean that specialists have recognized a problem as solved.

What is still needed to interpret and reproduce the evaluation

The result needs to be considered in light of its documentary status: the evidence provided comes from the paper’s initial arXiv version. The available information does not say that the work has passed peer review, been independently replicated or had its scores externally validated. That does not invalidate the benchmark, but it does mean its conclusions should be treated as preliminary results from the study described.

Assessing its strength requires reproducible details about the problem statements, instructions and rubrics, evaluated answers, criteria for what counts as progress, and agreement among evaluators. It would also be useful to know how automated judgments were checked against specialists or formal verification, and what steps were taken to address the possibility that problems or solutions appeared in training data. The available abstract does not resolve these questions. It therefore cannot establish either that contamination occurred or that it was ruled out.

An evaluation tool, not a certificate of discovery

OpenProblemBench proposes a way to measure progress on tasks where answers cannot easily be compared with a known solution. Its core elements are a set of open problems, contextual information, and four models evaluating correctness, completeness and progress. The approach addresses a real challenge for traditional benchmarks, but its results depend on the quality of the problem selection, the clarity of the checks and the reliability of the evaluators.

The most accurate conclusion is that the preprint reports promising results within its own protocol, with GPT-6-Astra leading among the configurations studied. Turning that signal into broader evidence will require verifiable details, expert assessment and independent replication. Until then, the benchmark is a proposal for investigating AI’s capabilities and limitations in foundational theoretical sciences, not proof that these systems have solved open problems in the scientific sense.

Keep exploring

Corrections and transparency

If you spot incorrect or outdated information, send us a correction with the page and source we should review.