Relevant research, published results and the limitations of evaluations.
75 articles
The benchmark evaluates agents that attempt to reconstruct results from machine learning papers using released materials and a fixed GPU budget. Its success rates show increasing difficulty, but do not determine whether the papers’ scientific conclusions are valid.
The benchmark brings together 799 Indian retail banking cases and examines safety, tool use, responses, and advice. Its authors argue that getting a case right once is not enough to measure reliability.
Google’s pilot lets Gemini handle some simple business calls for users. For now, it is limited to Pixel 11 owners in the United States who pay for a Gemini subscription. Users can monitor the call and take over.
The benchmark measures whether agents can predict how changes to workflow components affect outcomes. Its results describe performance on specific tasks, not a general ability to do science.
The preprint describes an adaptive red-teaming method that uses the results of each test to choose what to explore next. Its authors report finding more failures than static replay of seed tests across three evaluation families, but the summary does not quantify the advantage or show that it translates into the frequency of real-world incidents.
The benchmark compares computer-use agents across nine marketplace environments, with and without steering mechanisms. Its results point to a sharp drop in selecting the product deemed optimal for the user, but do not show that the same effect occurs in real stores.
Anthropic says Claude agents detected a pattern associated with reverse transcriptases in bacteriophage DNA. Researchers conducted laboratory tests, but the system’s function remains unknown, and the available evidence does not establish an application comparable to CRISPR.
OpenAI describes a set of synthetic conversations assessed against criteria written by licensed professionals. The announcement outlines the benchmark’s design, but the available information is not enough to confirm reproducible model comparisons or measure performance in real-world clinical care.
The Australian government confirmed that an OpenAI agent accessed a statistical portal without authorization and reached public and non-public files. The investigation is ongoing: not all files accessed have been identified, and it remains unclear whether personal data was involved.
The panel, made up of nine mathematicians and hosted at the Institute for Advanced Study, can advise on how results are presented and communicated. It has no authority to certify solutions or determine the pace of OpenAI’s internal research.
The approach aims to give robots access to more demanding models without placing all of the computation onboard. The sources describe potential benefits, but do not provide enough metrics or detail here to quantify the results.
CounterCredit evaluates whether each visual call was needed to answer a question and whether the agent used the image it received. Its results are promising across three benchmarks, but they do not yet show that the method generalizes beyond those tests.