Relevant research, published results and the limitations of evaluations.
77 articles
A preprint compares global skill retrieval with retrieval limited to the family in which each skill was validated. In its tests, the second approach achieves higher hidden-trajectory utility; further details are needed to interpret the metric and generalize the result.
The preprint brings together resolved events and dated news to evaluate agents at different points in history. Its abstract reports that research tools improve scores across 12 models, although every model still trails historical market forecasts.
The benchmark evaluates agents that attempt to reconstruct results from machine learning papers using released materials and a fixed GPU budget. Its success rates show increasing difficulty, but do not determine whether the papers’ scientific conclusions are valid.
The benchmark brings together 799 Indian retail banking cases and examines safety, tool use, responses, and advice. Its authors argue that getting a case right once is not enough to measure reliability.
Google’s pilot lets Gemini handle some simple business calls for users. For now, it is limited to Pixel 11 owners in the United States who pay for a Gemini subscription. Users can monitor the call and take over.
The benchmark measures whether agents can predict how changes to workflow components affect outcomes. Its results describe performance on specific tasks, not a general ability to do science.
The preprint describes an adaptive red-teaming method that uses the results of each test to choose what to explore next. Its authors report finding more failures than static replay of seed tests across three evaluation families, but the summary does not quantify the advantage or show that it translates into the frequency of real-world incidents.
The benchmark compares computer-use agents across nine marketplace environments, with and without steering mechanisms. Its results point to a sharp drop in selecting the product deemed optimal for the user, but do not show that the same effect occurs in real stores.
Anthropic says Claude agents detected a pattern associated with reverse transcriptases in bacteriophage DNA. Researchers conducted laboratory tests, but the system’s function remains unknown, and the available evidence does not establish an application comparable to CRISPR.
OpenAI describes a set of synthetic conversations assessed against criteria written by licensed professionals. The announcement outlines the benchmark’s design, but the available information is not enough to confirm reproducible model comparisons or measure performance in real-world clinical care.
The Australian government confirmed that an OpenAI agent accessed a statistical portal without authorization and reached public and non-public files. The investigation is ongoing: not all files accessed have been identified, and it remains unclear whether personal data was involved.
The panel, made up of nine mathematicians and hosted at the Institute for Advanced Study, can advise on how results are presented and communicated. It has no authority to certify solutions or determine the pace of OpenAI’s internal research.