Forecast-Dojo proposes a reproducible testbed for forecasting agents

A testbed for replaying forecasts about past events

Evaluating a forecasting system can be difficult when every test depends on events that have not yet been resolved. A future outcome may take time to become known, while the information available and the conditions of the evaluation can change in the meantime. Forecast-Dojo addresses this problem with an environment that reconstructs forecasting tasks about historical events and lets researchers evaluate a prediction again on successive dates.

The work is presented as a preprint and describes a reproducible environment for evaluating and training forecasting agents based on language models. It combines prediction-market questions whose outcomes are already known with dated news articles. An agent can therefore research an event and revisit its forecast at different points before the event was resolved.

The proposal does not amount to proof that an agent will forecast future events well in real time. Based on the available abstract, its main contribution is to provide repeatable historical tasks, research tools, and known outcomes against which forecasts can be compared.

How the timeline and available information are reconstructed

Forecast-Dojo brings together two elements: resolved events and news articles with dates. The timeline places each evaluation at a point in the past. Rather than asking only what a model would have forecast at the end, the environment allows researchers to see how its estimate changes as the event unfolds and new dated evidence becomes available.

The preprint’s abstract says that news published after the date of each forecast is restricted. This condition matters: if an agent could consult information published after the evaluation date, it would have clues that were not yet available at the simulated moment. The material provided here does not describe the precise technical filtering mechanism, so it is not possible to explain how the restriction is applied in each tool or query.

The dataset contains 1,568 Polymarket events, split by time into training and evaluation periods, and 18.8 million dated news articles. The abstract does not specify the date boundaries or how many events fall into each split. The figures therefore describe the resource’s overall scale, but are not enough to reconstruct the temporal partition in full.

What the evaluation of 12 models reports

The most prominent result in the abstract is that research tools lower the Brier score for all 12 evaluated models. The Brier score is an error metric for probabilistic forecasts: broadly speaking, a lower value represents a better result according to that metric. The abstract does not include the numerical scores, the size of the improvement, or uncertainty intervals, so it is not possible to assess the magnitude here or determine whether the differences are large.

The study also reports that forecasts improve as events unfold. The largest gains occur at steps where more new, dated evidence is recorded. This pattern is consistent with the idea that recent information can help update an estimate, but the abstract does not let us attribute all of the improvement to news alone. More detail would be needed about the comparison conditions and how the tools were used.

Another result puts the finding in perspective: every model trails historical market forecasts on both Brier score and accuracy. So the finding that tools help models does not mean that the models outperform the market benchmark used in the study. Nor does it show that every model improves by the same amount.

Tools, memory, and learning

The environment is intended not only to measure models, but also to collect agent interactions and provide feedback from outcomes that are already known. The preprint mentions supervised fine-tuning as a proof of concept. This points to a way of studying whether research trajectories and the consequences of predictions can be used during training, but the abstract does not quantify the effect of this procedure or establish it as a generally demonstrated improvement.

The work also evaluates a “belief notebook” that carries information between dates. According to the abstract, the notebook reduces research cost but does not consistently improve forecast quality. This is a practical distinction: a tool can save effort or help maintain continuity across a task without necessarily increasing predictive performance.

Interpreting these results depends on what counts as research cost, which specific tools were enabled, and how quality was measured. Those details are not in the abstract provided here. We should not infer that every memory system or search tool will have the same effects as the conditions evaluated in the study.

Limitations and open questions

The evaluation is based on a specific set of Polymarket events and a corpus of dated news. The selection of questions, the coverage of the articles, and the way the historical record is turned into tasks can all influence what the benchmark measures. The abstract provided does not explain these criteria in enough detail to judge whether the events represent a broad range of forecasting topics or whether some subjects are overrepresented.

The information available here also omits details needed to reproduce the comparisons independently: the names and versions of the 12 models, the exact tools, the instructions given to agents, the observed Brier scores, and the full details of the control conditions. The abstract confirms a temporal split and a restriction on news by forecast date, but that is not enough to verify all of these methodological elements.

The source is a preprint, not an independent validation. The available abstract helps explain the proposal and its main findings, but cannot by itself answer questions about robustness, sensitivity to event selection, or generalization to live forecasts. Independent evaluation and access to the data, code, and configurations would help establish reproducibility in practice.

What changes for people evaluating agents

Forecast-Dojo proposes a way to study forecasting agents without waiting for new events to resolve: reconstruct past tasks, control which information was dated as available at each point, and compare predictions with known outcomes. The reported result on tools is promising in a limited sense: the abstract reports a lower Brier score for all 12 evaluated models.

That conclusion needs to be kept alongside its limitations. The improvement does not erase the gap with historical market forecasts, memory does not consistently improve predictive quality, and the abstract does not provide the figures needed to estimate the effects’ magnitude. In addition, performance on historical data does not by itself establish how agents will handle new news or other domains.

For readers comparing tools and systems, the important news is not that models have already outperformed markets. It is that this work proposes a framework for repeating evaluations and studying the value of research with time-indexed information. Its ultimate usefulness will depend on whether methodological details, resources, and independent tests allow others to verify the conclusions and define their scope.

Keep exploring

Corrections and transparency

If you spot incorrect or outdated information, send us a correction with the page and source we should review.