Measuring agentic intelligence in the real world: inside Toloka Arena Enterprise Tool Use

An AI agent can complete a task and still get the job wrong. It can reach the expected outcome while skipping a required verification, ignoring a policy or taking an action it was never asked to take. In enterprise workflows, those steps matter as much as the result.

Toloka Arena’s Enterprise Tool-Use benchmark evaluates AI agents on multi-step enterprise workflows in simulated business environments. As described in the original introduction to Toloka Arena, each model works with mock systems, a written policy and a user who responds to what the agent does. Every task is attempted five times, and a model earns full credit only by passing all five.

Since launching in April, we have made the benchmark more demanding in three ways:

To understand what these changes add, we looked at ten models that ran on both the previous version and the current one. The two versions use different tasks, so the comparison shows how each model fares under each benchmark rather than on identical work.

The result is only half the story

Grading an agent is a little like a driving test. Arriving at the right destination isn’t enough; the examiner also checks how you got there. Every attempt now goes through two checks. An automatic check compares the final records with the expected result, and a language-model judge reviews the full run against the steps the task requires, such as verifying the customer, looking up records before acting and asking for confirmation where the policy calls for it. An attempt passes only if it clears both.

Grading matters. Gemini 3.1 Pro would score 29% on records alone but scores 8% once the steps count; in more than half its attempts it acts without first looking up the record. Among the weakest models, more than four in ten attempts break a required step; among the strongest, about one in ten.

The two checks also expose different weaknesses, enough to change the order at the top. For example, graded on the final records alone, Gemini 3.7 Flash would rank 5th on the leaderboard and GPT-6 Astra 11th; with the required steps included, Gemini 3.7 Flash falls to 13th and GPT-6 Astra rises to 5th.

Most importantly, it shows what a new model actually gets better at. On records alone, Fable 5.2 is barely ahead of Fable 5.1, at 80% against 78%. On the leaderboard, which also checks the required steps, the gap is three times larger, at 77% against 71%. Most of Fable 5.2’s progress is in how it does the work, not in the records it leaves behind.

The failure patterns show why. Fable 5.2 does steps out of order less than half as often as Fable 5.1, no longer breaks the conversation protocol, and leaks internal information less often. What hasn’t improved is classifying the case: overall it sets the category, priority, tags and flags wrongly as often as before.

How agents fail

A pass or fail tells you whether an agent succeeded; it doesn’t tell you why it didn’t. Every failed attempt is therefore classified into failure modes, grouped into three categories:

Each missed rule and each incorrect field is mapped to a mode using fixed rules, and a single attempt can show several. 

This breakdown changes the picture considerably. Under the old grading, every model’s most common failure was the same: an existing record left with the wrong value. With the process checks included, process failures take at least two of the top three positions for nine of the eleven models. Kimi K3, for instance, skips a required record lookup in six out of ten of its failed attempts. In other words, many of today’s most frequent failures were invisible to the previous version of the benchmark.

The failure-mode chart on the leaderboard shows each model’s failure modes as a share of its failed attempts, so you can see at a glance where a model goes wrong, or compare models side by side.

Eleven industries, more demanding workflows

The benchmark now covers 11 industries, up from seven. The four additions are airline cargo, pharma, retail brokerage and telecom, which join airlines, banking HR, logistics, manufacturing, restaurant operations and two travel businesses. Each is a simulated company with its own databases, tools and written policies.

The new environments lean toward regulated, process-heavy work, where following the correct procedure matters as much as reaching the correct outcome. That is exactly where a records-only benchmark would miss the most.

A more complete picture of reliability

Looking at the final result alone can make an agent appear more capable than it is. The process check shows whether it follows the policy, repeated attempts show whether it can do so consistently, and the failure breakdown shows where it falls short. Together, they give a clearer view of what an agent can be trusted to do in a real enterprise setting.

Explore the Enterprise Tool-Use leaderboard