The problem of obsolescence in agent memory
An artificial intelligence agent can commit a subtle but costly error: responding with information that was correct in the past but is no longer. If a customer states in March that their office is in Bangalore and communicates in June that they have moved to Pune, the agent must update that data. If it fails to do so and answers a location-dependent query in July as if the office were still in Bangalore, the consequence can range from simple friction to a serious operational error, such as booking catering for the old address.
This scenario illustrates the central difficulty of memory in long-duration agents. A reply built on an outdated fact reads exactly like one built on a current one, making the error invisible to the user and difficult to diagnose for the developer. The inability to distinguish between a storage failure and a usage failure is the main obstacle to improving the consistency of these systems.
A framework to measure real memory
Salesforce AI Research has developed an evaluation framework designed to answer a complex question: how much does an agent actually remember? This system generates long, multi-session conversations with specific facts planted over time. It then runs these conversations through the live Agentforce system and asks questions that depend on those facts.
The key to this approach is that it tests the full path, from the moment a customer mentions a detail to the moment the agent acts on it. This allows separating two types of failures: whether the agent did not save the information or whether it saved it but did not use it correctly. By identifying which of the two is the problem, engineers can improve the system by modifying the memory policy rather than rebuilding the underlying architecture.
Memory evaluation cycle
The evaluation process follows a logical sequence to validate the integrity of the agent's memory.
- 01Generation of multi-session conversations with planted facts.
- 02Execution of the conversations in the live Agentforce system.
- 03Formulation of questions that depend on the planted facts.
- 04Analysis of the response to distinguish between storage failure and usage failure.
- 05Adjustment of the memory policy based on the results.
The memory policy update
The performance improvement did not come from a change in the Agentforce Memory architecture, which continues to run on the same technical base as at launch. The change lies in the updated memory policy, which defines the rules for what information the agent retains and how it handles changes to that information.
The new policy has two fundamental characteristics: it retains more of what the customer says and treats an update as a correction to an existing data point rather than an additional fact filed next to the first. This paradigm shift allows the agent to overwrite outdated information with new data, maintaining context consistency over time.
Comparison of memory policies
Key differences between the previous and updated policies in managing customer data.
| Feature | Previous Policy | Updated Policy |
|---|---|---|
| Update management | Archives the new data next to the old | Treats the new data as a correction to the old |
| Information retention | Retains a subset of data | Retains more of what the customer says |
| Temporal consistency | May respond with outdated data | Keeps information updated |
Performance impact: duplication metrics
The results of the evaluation on the live system show a significant improvement. The updated policy recalls 1.8 times more of the facts that customers state explicitly than the policy it replaced. In the case of facts that change over time, the improvement is even greater, with a recall 2.2 times higher.
These figures indicate that the end-to-end conversation memory has doubled in quality. This is not just a statistical improvement, but a tangible change in the user experience, as agents are now more reliable for maintaining context over months of interaction, reducing the need for customers to repeat information and minimizing operational errors.
Implications for development and governance
This update underscores the importance of measuring AI agent performance under real conditions, not just in laboratory environments. As Silvio Savarese, EVP and chief scientist at Salesforce, noted, in a product organization, the problem is already running with real customers, and the discipline is to measure it honestly before making changes. This approach allows development teams to identify and correct specific failures without needing to redesign the infrastructure.
For companies using Agentforce, this improvement means greater confidence in the agents' ability to handle complex and changing contexts. Agent memory is becoming a standard that businesses demand, similar to any other part of their systems. The ability to test what an agent remembers across a full conversation is how that standard gets set.
Keep exploring
Sources consulted
Corrections and transparency
If you spot incorrect or outdated information, send us a correction with the page and source we should review.
Submit a correction