Step-by-Step Verifiers: What Process Reward Models Show—and How Far They Generalize

The question: does better selection mean reliable reasoning?

When a model solves a problem in several steps, there is more than one way to evaluate its answer. We can judge only the conclusion, or inspect the intermediate steps and estimate whether each one is valid. The second approach appears to offer a more detailed diagnosis: if we can identify where an error begins, in principle we can reject a solution before that error affects the rest of the reasoning.

A Process Reward Model (PRM) is a model trained to assign scores to steps in a solution, usually to distinguish correct from incorrect steps, or useful steps from defective ones. An Outcome Reward Model (ORM), by contrast, evaluates the complete result. Both are evaluators; neither necessarily generates the solution. If they are used to rank several candidate answers, they participate in a search process. That search is an additional operation, not something established by the score itself.

The conclusion supported by the studies considered here is limited: process verifiers can help select solutions in particular tasks and experimental conditions. That is not enough to conclude that every scored step is correct, that a visible explanation faithfully reproduces the model’s internal process, or that the verifier will behave the same way in another domain. It is important to distinguish the result on a specific task from broader claims about reliability.

Four things not to conflate

Process supervision provides labels for intermediate steps. Outcome supervision provides labels for complete solutions or their final answers. A PRM learns from signals about the process; an ORM learns from signals about the outcome. A general-purpose critic can receive a solution and produce an evaluation in natural language without necessarily being a PRM trained on step-level labels.

Verification and guided search are not equivalent, either. Verification means assigning a judgment or score to a solution or its steps. In search, a generator produces alternatives and a selection rule decides which to keep or explore. A PRM can serve as that rule, but the final performance also depends on the generator, the number of candidates, the search procedure, and the computational budget.

These distinctions matter when interpreting results. If a PRM-based configuration produces more correct answers than an ORM-based one, the finding may support the PRM’s usefulness for that particular combination of task, model, and search procedure. It does not automatically establish that the PRM is better at detecting errors, or that the improvement comes solely from understanding the reasoning.

Let’s Verify Step by Step: evidence from mathematical problems

Let’s Verify Step by Step studies process supervision versus outcome supervision in mathematical reasoning. The paper introduces PRM800K, a dataset containing 800,000 step-level correctness labels for model solutions to MATH problems. The annotation unit matters: these labels make it possible to train and evaluate intermediate judgments on this type of material, but they are not a universal collection of rules for reasoning.

The paper reports that, in its mathematical problem experiments, process supervision can improve solution selection compared with outcome supervision. That result should be read within the paper’s experimental setup: MATH problems, candidate solutions generated by models, and verifiers trained using the data and procedures described. It does not show that every PRM outperforms every ORM with every generator or on every task.

There is also a difference between selecting the right answer and judging every step correctly. If a verifier ranks solutions well on a test, that measures its usefulness as a selector on that test. To claim that it reliably locates errors, we would need a direct evaluation of its ability to identify correct and incorrect steps, using appropriate references. An aggregate score on final answers cannot substitute for that analysis.

ProcessBench: evaluating where the first error occurs

ProcessBench addresses a more direct question about step evaluation: given a line of reasoning containing errors, can an evaluator identify the first incorrect step? The benchmark contains 3,400 cases and uses specialist human annotations to identify the first error. Its design makes it possible to study something different from simply selecting a final answer: locating an error in a reasoning sequence.

The paper compares PRMs with critic models and reports results on mathematical reasoning tasks. These comparisons provide a shared test for the systems included in the benchmark, but their scope is bounded by the tasks, solutions, and annotation criteria it contains. A strong ProcessBench score would support performance on that evaluation; it would not demonstrate the same capability in code, science, or other kinds of reasoning.

Annotating the first error does not answer every possible question about a solution, either. Annotators may reasonably disagree about how finely a step should be divided; there may be transcription errors; or a step may appear incorrect even though the later conclusion is correct by another route. So, in addition to citing an overall score, it is worth checking how the evaluated units are defined, what instructions annotators receive, and how disagreements are resolved.

The project’s official repository documents access to the benchmark and its resources. For a reproduction, having the benchmark’s name is not enough: record which data version, input format, model, instructions, and configuration were used.

Rewarding Progress and ThinkPRM: other ways to build and test verifiers

Rewarding Progress introduces Process Advantage Verifiers and studies their use in reasoning tasks, including guided search and reinforcement learning with verifiers. Its relevance here is that it shifts attention from “Which solution has the highest score?” to how a progress signal can be used in procedures that generate or select solutions. The paper compares process- and outcome-verifier configurations, but those comparisons apply to the models, tasks, and budgets defined by its experiments.

A guided-search result combines several decisions: which candidates the model produces, which signal the verifier uses, how many iterations are run, and how much compute is allowed. If the rate of correct solutions increases, that improvement belongs to the complete configuration. To attribute it to the verifier in particular, an experiment must control the other variables and report costs, not just final quality.

ThinkPRM studies process verifiers that generate reasoning about their evaluation of steps. The paper reports evaluations on ProcessBench, MATH-500, and AIME ’24, as well as out-of-domain tests on GPQA and LiveCodeBench. This broadens the evidence beyond a single mathematical test, but it does not turn those results into a guarantee of open-ended generalization. Each dataset represents specific coverage, and performance may vary with the task, distribution, and presentation of the steps.

A method’s name alone does not determine how much or what kind of supervision it requires, or which baseline is appropriate. To compare these studies, extract the training set, labels used, generator and evaluator models, instructions, budgets, and metrics from each paper. When those details are not identical, a simple ranking of “winners” would be misleading.

Limits: costly labels, judgment errors, and proxy overoptimization

Step-level supervision may be more informative than a final-answer label, but obtaining it requires deciding what counts as a step and whether that step is correct. PRM800K shows the scale of a resource devoted to step labels for mathematical problems; that scale does not eliminate the cost of producing, reviewing, and maintaining reliable labels. Nor does an annotation convention for mathematical solutions automatically transfer to tasks with less discrete criteria.

A verifier can also make mistakes. A false positive accepts a defective step; a false negative rejects a valid one. If the system selects among many answers, these errors do not necessarily have symmetric effects: a mistaken high score can promote an incorrect solution. It is therefore useful to measure both kinds of error and review examples, rather than relying only on an average metric.

Proxy overoptimization is another practical risk. If the generator is repeatedly optimized to obtain high scores from a verifier, it may learn patterns that the evaluator rewards without improving actual correctness. This does not mean that it always happens, but it is a possibility worth investigating: compare the PRM score with independent checks, audit selected solutions, and look for cases where a persuasive trace or superficial phrasing receives a high score despite being defective.

Finally, scoring written reasoning does not show that the text is a faithful transcript of the model’s internal calculations. Experiments on selection or error localization evaluate observable behavior under defined tasks. They are not enough to establish internal transparency or to claim that adding a PRM makes an application safe.

A practical protocol for evaluating your own PRM

A team considering a PRM can start with a small, controlled test without confusing an evaluator comparison with a complete product evaluation. Define the objective before running the experiment: do you want to select better answers, detect the first error, reduce calls to a critic model, or improve a search process? Each objective calls for different data and metrics.

To compare systems, use a frozen task set, including cases that were not used to tune the instructions, and references verified by people or independent methods. Keep the same generator models and candidates when the goal is to compare evaluators, and hold the selection budget constant. If the PRM uses more calls or allows more exploration than the baseline, the comparison should report that difference.

The system set should include at least a PRM, a general-purpose judge, and an outcome verifier. The general-purpose judge evaluates the solution using a natural-language instruction; the final-answer verifier checks only the conclusion, when the task allows for reliable checking. Neither is a universal control: they help establish whether step-level information improves the judgment relative to specific alternatives.

In addition to the primary metric, record correct and incorrect judgments separately, first-error localization quality where relevant, cost per candidate, number of candidates, and the rate of correct solutions as the budget varies. Reserve some tasks for domain or difficulty shifts, and manually audit a sample of cases where the PRM and baselines disagree.

Defensible conclusions

Research on PRMs provides evidence that evaluating intermediate steps can be useful for selecting solutions and, on benchmarks designed for that purpose, for studying error detection. Let’s Verify Step by Step documents process supervision on mathematical problems; ProcessBench formalizes an evaluation focused on locating the first error; Rewarding Progress explores verifiers within search and learning procedures; and ThinkPRM extends testing to mathematical datasets and specific out-of-domain evaluations.

The conclusion is not that there is a universal winner. Tasks, data, labels, models, budgets, and success criteria differ. The most rigorous claim is conditional: a PRM may add value in an evaluated configuration, and that value should be measured against comparable baselines with an explicit analysis of costs and errors.

To decide whether a PRM is useful in a particular workflow, it is not enough to ask whether the model assigns convincing scores. Measure whether those scores improve the operational objective, check which cases it gets wrong, test distribution shifts, and keep answer correctness, step validity, and the fidelity of visible reasoning separate. That separation makes the evidence more modest, but also more useful.

Keep exploring

Corrections and transparency

If you spot incorrect or outdated information, send us a correction with the page and source we should review.