Sora 2 Pro in Video Benchmarks: What Blind Preference Measures—and What It Leaves Unanswered

A video score is not a complete evaluation

Saying that Sora 2 Pro “performs well” on a video benchmark can refer to different results. It might mean that people preferred its clips in a pairwise comparison, that it scored highly on an automated dimension, or that it followed a set of instructions more successfully. These conclusions are not interchangeable: each depends on what was measured, which samples were used, and under what conditions.

This distinction matters because an aggregate ranking can summarize an overall preference without diagnosing which property of a video caused that preference. A clip might be more appealing in a vote while also containing continuity errors, omitting elements from the prompt, or poorly synchronizing its sound. To attribute a specific strength, consult the relevant task and metric rather than inferring it from a position in a ranking.

The available sources make it possible to describe the methods used by several evaluations, some results published by an arena, and the options available in the Sora API. They do not show that Sora 2 Pro was evaluated on the VBench or Video-Bench benchmarks. Those studies are useful here for explaining which dimensions a benchmark can separate; they should not be presented as results for the model.

First, identify what was actually evaluated

“Sora 2 Pro” is not enough information to describe an experiment. A useful report should specify the exact variant, the access channel, and the generation date. It should also clarify whether samples were obtained through the API or another interface, which options were available, and whether any post-processing took place. Without those details, an observed difference cannot safely be attributed to the model alone.

OpenAI’s documentation identifies sora-2-pro and describes parameters such as size and duration, as well as the option to use a reference image. That makes it possible to ask specific questions when reading a result: Was text-to-video or image-to-video used? What size was requested? How long was the clip? Was audio generated? The fact that an option is documented does not prove that a benchmark used it; the protocol must say so.

It is also worth recording the date because systems, interfaces, and access conditions can change. Arena’s changelog documents the addition of Sora 2 and Sora 2 Pro to its text-to-video leaderboard, but a record of when a system was added does not, by itself, describe all generation details or guarantee that leaderboard results from different periods are directly comparable.

What a preference arena can tell you

A preference-based arena typically presents outputs for someone to choose between, without deliberately letting the system’s name determine the answer. In a blind comparison, the choice provides evidence about preference in that format: which clip the participants liked more among the samples they saw. This is a relevant signal about human perception, but its interpretation should remain within those bounds.

Arena’s text-to-video leaderboard displays Sora 2 Pro with a score and an interval, along with a stated cutoff date. The responsible way to report this is to attribute the result to that leaderboard and that snapshot, not to describe it as a universal measure of quality. The interval represents estimation uncertainty under the platform’s method; it should not be omitted or turned into a guarantee that one system will be superior for every prompt.

Arena’s ranking-method documentation explains how scores are calculated and how uncertainty and ranks are handled. Even so, assessing the strength of a claim requires checking which comparisons were included, how they were assigned, how many observations support the result, and which prompts were represented. If the page does not provide all those details in the view consulted, they should not be filled in with guesses.

A blind preference also does not identify the reason for a choice on its own. People may favor composition, style, readable motion, or apparent prompt fit, and several factors can come into play at once. A vote is not automatically a rubric for instruction following, temporal continuity, or audiovisual synchronization.

Why dimension-specific benchmarks matter

Benchmarks built around explicit tasks or dimensions let us ask which aspect is being evaluated. VBench, for example, organizes video-generation evaluation into multiple dimensions, including consistency, temporal flicker, and motion smoothness. Video-Bench describes itself as a benchmark aligned with human judgments. These designs illustrate alternatives to reducing evaluation to a single overall preference, but they are not evidence that Sora 2 Pro achieved any particular result on those benchmarks.

Separating dimensions offers diagnostic value. If an evaluation finds a weakness in temporal consistency, that does not necessarily mean overall visual quality is low. If a model performs strongly on a visual dimension, that does not prove it respects every detail in an instruction. Metrics should be interpreted according to the benchmark’s definitions and procedures, and an automated score should not simply be equated with a human judgment.

Comparison with an arena can be complementary. A preference test summarizes which output is chosen under a given protocol, while a set of dimension-specific tests aims to describe separate properties. Neither is sufficient for every question: the first does not automatically provide a causal diagnosis, and the second may not reflect which clips people prefer in a particular use case.

Comparability: prompts, generations, and clip selection

Two results are comparable only if we know which conditions they share. Input modality is one initial point of difference: generating from text is not exactly the same task as animating a reference image. Requested duration and size, whether audio is present, and any constraints applied to each system also matter. Interface documentation can confirm which parameters are supported, but not which ones an experiment used.

The prompt is part of the protocol, not an editorial detail. A collection of simple instructions, a list of complex scenes, and a set requiring actions in a precise order can produce different performance profiles. If a benchmark does not publish or describe its inputs, it becomes harder to understand what its sample covers and to repeat the test.

The number of generations per prompt and the selection rule are especially important. If several outputs are generated and one is chosen for publication, the result also reflects the selection process, not just an individual generation. A fair comparison should state whether the first output was shown, whether an output was selected according to a predefined rule, or whether another sampling method was used. The sources provided do not establish a single rule that applies to every Sora 2 Pro evaluation; this must be verified case by case.

Human evaluation can also vary: who votes, what instructions they receive, and which elements they can see. If audio is included, the report should specify whether voters consider the sound or only the image. If automated evaluators are used, the report should describe what they assess and how their use was validated. Without such details, a number may still be informative as a published result, but its scope is narrower.

Attribution limits: model, interface, and configuration

An evaluation does not observe a model in the abstract: it observes outputs produced through a particular system and under particular conditions. The result may depend on the variant, parameters, access method, generation policy, and any selection or post-processing. If services with different interfaces are compared, a difference does not automatically isolate the effect of the model.

It is therefore useful to distinguish three levels when writing about a result: the observed outcome (“these samples received these preferences”); a compatible interpretation (“in this test, under these conditions, one output was chosen more often”); and a general claim (“the system is better”). The third level requires more evidence than the first or second: diverse tasks, comparable conditions, and results that support the specific property being attributed.

The VBench repository and the Video-Bench papers offer methodological points of reference for understanding multidimensional evaluations and their artifacts. The VBench repository brings together code, prompts, dimensions, evaluation instructions, and links to videos and configurations. The existence of reproducibility materials for that project does not mean that prompts, generations, or configuration from an Arena evaluation are available. Reproducibility must be checked separately for each result.

Reproducibility after the API shutdown

OpenAI says the Sora API and models will be shut down on September 24, 2026. Its help documentation distinguishes that shutdown from the end of the web and app experiences. The announcement should therefore not be described as if every form of access will necessarily end at the same time. The date refers to the API shutdown according to the provider’s documentation; anyone attempting a future reproduction will need to check the conditions in effect and whether access is actually available.

The methodological consequence is clear: an archived result can remain a useful observation even if it is no longer possible to request a new generation with the same configuration. Re-scoring stored clips is also not the same as repeating the generation experiment. In the first case, existing material can be evaluated again; in the second, access to the generation process, configuration, and original inputs is required.

A historical report should state what has been preserved: videos, prompts, parameters, dates, evaluation instructions, disaggregated results, and aggregation procedure. If only a leaderboard or published score remains, it can be cited as a record of that page, but it cannot be reconstructed as a complete test. The available sources do not confirm that all these artifacts exist for the Sora 2 Pro leaderboard; their availability should be treated as uncertain, not assumed.

A checklist for reading results about Sora 2 Pro

Before turning a ranking into an editorial or technical conclusion, check exactly which claim it supports. The benchmark name, score, and position in a table are not enough if the task, samples, and method are unknown. An incomplete record can still be cited cautiously, provided the omissions are clear and the result is not credited with measuring a capability it does not test.

In particular, avoid directly comparing a preference ranking with a dimension-specific benchmark metric. Also avoid comparing different periods within the same arena without checking for methodological changes. Do not present results from different models as if they had the same prompts, parameters, and generation opportunities when the source does not document that they did.

The strongest conclusion the sources support is methodological: a preference can indicate which sample people liked more under a particular protocol; a set of explicit tasks can investigate bounded properties; and reproducibility depends on artifacts and access that must be verified. None of these, on its own, warrants a universal claim about Sora 2 Pro’s quality.

Keep exploring

Corrections and transparency

If you spot incorrect or outdated information, send us a correction with the page and source we should review.