Exact version
Check the model identifier, date and whether the alias can change.
Published results and conditions declared by their sources. Figures are comparable only when version, configuration, metric and date match.
| Model | Organization | Result | Metric | Conditions |
|---|---|---|---|---|
| Claude Haiku 4.5 | Anthropic | 73,3 % | Resolved | Average of 50 trials; scaffold described by Anthropic. |
| Mistral Medium 3.5 | Mistral AI | 77,6 % | Resolved | Result published by Mistral. |
| GPT‑4.1 | OpenAI | 54,6 % | Published solved problems | Result or evaluation published by the provider; review methodology and conditions in the source. |
| Qwen3‑Coder 480B A35B | Alibaba Qwen | Published open SOTA result | Software problems solved | Result or evaluation published by the provider; review methodology and conditions in the source. |
| Recursive Flint | Recursive | 74.2% | Higher is better | Recursive · mean pass@1 over at least five attempts · temperature 1.0, top-p 0.95, top-k 20. Terminus 2 for Terminal-Bench; mini-swe-agent for SWE-bench. Provider result, not reproduced by Inferama; do not extrapolate between precisions. |
| Nemotron Labs Dory 1.0 4B | NVIDIA | 59,5 | BF16 score · 2 loops | NVIDIA · BF16 · 1 / 2 / 4 loops: 50,3 / 59,5 / 61,8. NeMo Gym. Not reproduced by Inferama. |
Check the model identifier, date and whether the alias can change.
Review tools, effort, number of attempts, prompt and inference budget.
Validate the result with tasks, languages, formats and errors representative of your product.
A public benchmark provides guidance, but does not replace an evaluation with your data, tools, budget, and error tolerance.
A moving alias and a snapshot may produce different results. Record the model, date, and provider.
Prompt, reasoning, tools, number of attempts, and budget must be equivalent for comparison.
A small difference may disappear between runs. Keep samples, dispersion, and failures, not just the average.
Validate whether the improvement holds across languages, formats, and real cases close to your product.
A metric is only useful when we know the task, the conditions, and the error that matters. The result begins with the design of the test.
A model performing better alongside a component does not prove that the component caused the improvement. In AI studies, causal attribution requires defining the comparison, accounting for factors that could also affect the result and, where possible, intervening on the component. Observational inference can be informative, but it depends on assumptions that should be made explicit.
15 Nov 2026 ↗ ANALISISA cost figure per resolved issue is useful only if it reveals everything that happened before the patch was obtained: failed attempts, token consumption, stopping rules, selection among runs, and evaluation resources. This analysis proposes a reproducible scorecard for reading and comparing SWE-bench results, with particular attention to SWE-bench Verified.
22 Sep 2026 ↗