Toloka and xmemory test what memory does for AI agents

Once you've worked through a task a few times, you'd hope to have learned something that makes the next attempt easier. Toloka and xmemory put the idea to the test across 11 AI models and about 20,000 graded runs on Toloka’s enterprise tool-use data. The work showed where stored lessons helped agents improve, while the evaluation caught mistakes that would have been easy to miss by looking at overall scores alone.

A company policy tells an agent what it's allowed to do, and advice from previous attempts can help it avoid a mistake someone has already made. Making knowledge available when agents need it is what xmemory is built for, with its memory system organizing information into connected records that agents can query as they work.

To find out how much memory systems help, the team needed to see what the agents had done after reading the advice. A customer request might involve changing a booking or updating an account, and the correct steps still had to produce the right records.

Toloka and xmemory partnered to find a way to compare performance with ordinary document search to see what changed when agents could consult lessons from earlier work.

The approach

Toloka supplied the evaluation environment from its off-the-shelf enterprise tool-use dataset: support desks where agents handle customer requests across CRM, billing, and case-management systems. Each task has a verified reference state for the database, so success means the right records were written, not just the right things said. xmemory supplied the memory layer.

Pilots ran across several of these desks, and the full evaluation focused on airline cargo, where structured memory was the best fit. Agents there handled requests such as priority changes and damage claims.

During those pilots, both teams refined their side of the setup, with Toloka narrowing how tools were described to agents, and xmemory adding declared relationships between the record types in its store.

Agents spoke to customers and worked through requests using tools connected to the desk's systems. TolokaForge, Toloka's open-source engine for running and grading agents, tested each model on the same 50 tasks, with five attempts per task in each configuration.

Agents looked up company policy in one of two ways: a standard document search or xmemory's structured store. Both were loaded with the same nine policy articles, so they held the same information. 

The next question was whether advice from earlier attempts could help agents make better use of what they knew. The advice came from a separate analyst agent, which reviewed graded attempts on 35 tasks and wrote lessons based on what had gone right or wrong.

The remaining 15 tasks, drawn from the same request families, were held out from the analyst. They gave the team a way to see whether the advice helped with requests it hadn't been written from. An additional test served earlier lessons through document search on three models, so the team could examine the contribution of the retrieval engine.

Each attempt was cross referenced against the job requirements. Half the score came from an LLM's assessment of the procedure followed, and the other from examining the records left in the database. Toloka kept both results available separately, which allowed the team to identify an agent that followed more of the expected steps but still completed the request incorrectly.

The findings

All 11 models matched or exceeded baseline scores

Combining xmemory's policy retrieval with analyst memory brought every model to at least the level of document search alone, although the three smallest gains were within statistical noise. Lessons helped most below the top tier because the strongest models already followed the process well and had less to gain.

Prepared guidance also helped a mid-size open-weight model compete with proprietary alternatives. Qwen3.8 27B scored 0.75 with lessons, compared with 0.69 for GPT-5.6 Terra and 0.68 for Sonnet 5 without them. The comparison reflects different configurations, showing how access to advice can affect the result.

xmemory plus analyst memory versus document search alone across all 50 tasks, including the 35 used to create lessons. Cost is ratio to baseline; 0.90× means 10% lower.

Seven models ended up cheaper with the combined setup, and another two stayed within 3% of baseline cost. Grok 4.6 and GPT-5.6 Terra read enough additional memory text to increase their bills. Every model took longer to finish in the version tested, which gave the team a fuller picture of what the score improvements cost in practice.

Lessons improved performance beyond the tasks they were written from

Lessons from graded attempts raised scores for 10 of the 11 models on the tasks used to write them. One lesson, for example, reminded agents to look at whether a case already existed before opening another. The benefit also extended to held-out work, with six models scoring higher on the 15 held-out tasks and four of those gains having confidence intervals above zero. Average scores rose by 0.084 on the tasks used to write the lessons and 0.048 on held-out tasks.

Changes after adding analyst memory, with xmemory supplying policy in both configurations. Database and process figures are weighted contributions to the overall score change.

Bars show the range the true gain likely falls in; bars crossing zero mean no clear improvement.

The higher scores didn't always reflect how accurately the agents had completed the work when Toloka examined the database records. Agents followed procedures more closely on held-out tasks, but the records they left behind didn't improve. Sometimes a playbook written for one type of request was applied to another where the advice didn't fit. Examining the completed work helped the team see where a lesson needed more care before it could be relied on.

Evidence of what drove the improvement

The lessons, not the retrieval engine, drove the gains. When an earlier version of the same lessons was served through document search on three models, scores rose about as much. Final scores were about 0.01 to 0.02 apart. In a separate pilot, agents wrote their own memories without feedback on whether their earlier work was correct, but performance didn’t improve.

In a separate pilot, agents wrote their own memories without feedback on whether their work was correct, and performance didn’t improve. What made the difference was grading: the analyst learned from runs scored against verified outcomes in Toloka’s data.

A 26% reduction in the measured bill

Swapping document search for xmemory's policy store reduced the measured bill by 26% with no significant change in quality. Document search returned whole articles, averaging 26 KB to 65 KB per read, while xmemory typically supplied 2 KB to 9 KB of records. Agents read 3.5 to 15 times less policy text for the same output. 

The shorter answers remained smaller as the conversation continued and reduced the text billed on later turns. Across the models, prompt tokens fell by 30%. Every model became cheaper under the policy-store comparison.

The lower bill came with little change in average quality: the score difference of −0.008 had a 95% confidence interval spanning zero. Measuring latency wasn’t a goal of this study. xmemory updated its engine during the project, so read times varied between runs, and the team continues to work on read speed. For that reason latency figures are not reported here. In general, the benefits of a memory layer can come with slower responses, which is worth taking into account in a real implementation.

Models also made different use of the same policy store. Grok 4.6 responded to compact answers by asking more focused questions and reached parity with document search. Kimi K3 kept asking broad questions and missed relevant information. Testing the models individually showed that the same memory doesn't work equally well for every agent: how an agent asks its questions shapes what it gets back.

Lessons from the collaboration

A better score only tells you so much. Grading the process and final records separately helped both teams see why an agent had improved and whether it had completed the customer’s request correctly. 

xmemory could see where structured policy retrieval reduced costs and when agents benefited from stored lessons. Toloka could see improvements that looked more convincing in a single score than they did under closer scrutiny, along with patterns in the mistakes agents were still making.

The next evaluation

The study tested short, single sessions against a fixed set of policies, which isn't the long-running, constantly changing work xmemory is designed for. The next step would bring changing information into the evaluation, with agents responding to updates while a task is underway. Toloka's approach shows whether memory helps agents keep decisions current as circumstances change.

Request samples of Toloka’s enterprise tool-use data to test your agents against the work they’ll actually do.