← Blog

/

Insights

Insights

What the scores don't tell you about AI agents on Toloka Arena

Toloka Arena is live. See how your model ranks.

Closing a support case doesn’t count for much if the agent updated the wrong account or didn’t process a refund. Yet an AI judge can give the conversation a passing rating because the agent followed the required steps, while a separate automated assessment finds the records are wrong.

In Toloka’s enterprise RL Gyms, more than  half of failed attempts among the top 10 models passed the conversation assessment. The failure modes now available on Toloka Arena show where individual models go wrong so you have a clearer idea of what needs fixing.

At a glance

  • Every model on Toloka Arena now has a failure profile across 10 categories .

  • Among the top 10 models, about half of failed attempts looked fine in the conversation but left the wrong data in the system.

  • Compare models in the public chart, then request a write-up on your own model to understand its recurring mistakes.

  • The best model gets 77% of tasks right on all five attempts. Toloka's own training runs show how much task success can improve.

Meet the new chart

Pick a model to see which mistakes show up most often, or compare up to three side by side. Models sitting close together on the leaderboard can leave you with very different problems. The chart solves the problem by giving you a way to compare those problems before committing to a model.

Toloka Arena leaderboard chart, using the latest full evaluation for each model.

The 10 failure modes

Each category covers a mistake an operations lead would flag in a human agent. Rates show the share of all attempts affected, so lower is better. A single attempt can include several failure modes.

Category

What goes wrong

Everyday equivalent

Typical model

Worst model

Procedure adherence

Skips a required step, does steps out of order, misses an escalation or breaks the conversation protocol

A pharmacist hands over the medicine before checking the prescription (new)

19%

74%

Case status handling

Opens, resolves or closes the case at the wrong point, or leaves it in the wrong status

A waiter clears your plate before you've finished

18%

87%

Case classification

Sets the wrong category, priority, tags or flags

A mail room puts an urgent parcel in the "whenever" pile (new)

14%

75%

Information gathering

Acts without first checking records, history or policy

Buying milk without checking the fridge, which is already full of it

13%

92%

Record accuracy

Creates duplicates, misses records, links the wrong one, or writes wrong values

A clerk files your form under your neighbor's name (new)

13%

83%

Case documentation

Leaves the case notes, subject or description incomplete or inaccurate

A builder finishes the job and leaves no notes for the next crew

12%

67%

User communication

Does not inform, explain, confirm or ask consent when it should

A mechanic fixes your car and never calls to say it's ready

11%

51%

Confidentiality

Shares internal or confidential information with the user

A shop assistant reads you the manager's private notes about you

10%

54%

Identity & eligibility checks

Skips or delays confirming who the user is or what they are entitled to

A locksmith opens any door for whoever says it's their house

6%

30%

Honesty & commitments

Tells the user something false, or promises what it should not

A friend says "I posted your letter" while it's still in their bag

5%

33%

Model by model: what the score doesn't tell you

Two models a few points apart on the leaderboard can fail in completely different ways. Their failure profiles show what you're taking on with each.

Fable 5.2: the new leader, with the quietest failures 

Fable 5.2 gets 77% of tasks right on all five attempts and 88% on a single try, the highest on the board. Its lead over Opus 5.5 is within the margin of error. It fails 12% of attempts, fewer than any other model, and about half of those failures pass the conversation assessment with wrong records. That's 6% of all its attempts, the lowest share among the top 10. When it does fail, the most common category is case classification.

Opus 5.5: second on the board, with a blind spot

Opus 5.5 gets 75% of tasks right on all five attempts, yet roughly one in 12 of its attempts leaves records wrong behind a passing conversation rating.

GPT-6 Astra: the mistakes are in the records

Only 2% of Astra's failures are confined to the conversation, the lowest share on the board. Almost all the rest involve wrong records. And while Astra trails Opus 5.5 by just six percentage points on a single try, it leaves wrong records behind a passing conversation twice as often: in 16% of all attempts, against 8%. GPT-6 Sol and GPT-5.6 Sol share a similar pattern.

Gemini 4 Argon: all or nothing

Of the tasks Argon solves at least once, it gets 83% right on all five attempts, the best proportion on the board. But it fails every attempt at 12% of tasks, twice as many as Fable 5.2 or Opus 5.5. The models also succeed on different tasks: between them, at least one gets all five attempts right on 85% of tasks, compared with 75% for Opus alone. Argon and Fable 5.2 together also reach 85%.

GPT-6 Sol and Grok 4.6: the coin-flip models

Both score 76% to 77% on a single try, then lose about 20 percentage points when the task has to succeed on all five attempts, the steepest drop among the 15 models scoring at least 55%. Getting a task right once is still a long way from being able to rely on them.

Gemini 3.7 Flash: right records, wrong conversation

In 45% of Gemini 3.7 Flash's failures, the records end up correct but the agent breaks a required step with the user. That's the highest share of conversation-only failures on the board, and a very different problem from the one facing teams using the GPT models.

Internal information slips into the conversation

Several models in the middle of the board share information meant for internal use. Qwen 3.8 Max does so in 14% of all attempts, rising to 18% for Mistral Large 4. DeepSeek V4.1 Flash reaches 20%, while Haiku 5.5 reaches 22%. The rates are roughly one and a half to two and a half times the median.

Some models act before looking up the record

Granite 4.2, Mellum 2.1 and LFM2.5 skip a required record lookup in roughly six to nine out of 10 attempts. Across the same models, 84% to 92% of failures affect both the conversation and the records. 

The chart gives you the pattern. In a write-up on your own model, we examine failed attempts to explain the behavior behind it.

What the failures reveal

The better the model, the quieter its failures

Near the bottom of the leaderboard, a failed attempt usually goes wrong in both places. Among the 10 lowest-scoring models, 65% of failures affect the conversation and the records, while only 20% leave wrong records behind a passing conversation.

Move up the leaderboard and the balance changes. In the middle, 43% of failures pass the conversation assessment despite incorrect records. Among the top 10, it's about half. Better models fail less often, but more of their failures escape the conversation rating.

We reviewed a sample of those failures. About two-thirds were clear model errors, such as a required tag left off or a record linked without being requested. The rest needed a closer look at the task or grading. The broader pattern appeared model by model and in a second, independent analysis of the same runs.

Almost every task has been mastered by some model. None has mastered them all.

The best model solves three in four tasks on all five attempts. Across the board, however, 97% of tasks have been completed on all five attempts by at least one model, and every task has been solved at least once. Many of the tasks still troubling a leading model are already being completed consistently elsewhere on the board. There's good reason to keep working on them.

Upgrades can trade one failure for another

In two of the six upgrades we tracked within model families, a particular failure became three to four times more common even as the overall score rose. Required records went uncreated, or agents looped until they ran out of turns. A score tells you a model improved. The failure breakdown shows where it got worse.

A good score is only the start

An enterprise needs an agent that can repeat a successful result. Among models scoring at least 60% on Toloka Arena, the difference between doing a task once and doing it consistently is still considerable.

  • The median success rate is 80% on a single try, falling to 66.5% when all five attempts have to succeed. Repeating the task reveals uncertainty a single run won't capture.

  • About one in five tasks produce a mixture of passes and failures. Teams can use those mixed results to investigate why an agent succeeds on one attempt and misses the next.

  • The two scores are also more tightly linked near the top. Among these models, each extra percentage point of single-try success is associated with about 1.4 points of success across all five attempts, against about 0.75 for the rest of the board.

Training for the next successful attempt

We've put the enterprise environments behind Toloka Arena to use in our own training runs. After just one epoch, Qwen3.5-27B doubled how often it completed a task successfully three times in a row, from 23% to 46%. Success on held-out tasks rose from 41.6% to 64.5%. Read the Qwen3.5-27B results.

In a separate experiment, a 4B model improved from 6% to 71% task success. Read the 4B results. The environments are available as off-the-shelf enterprise RL data, so model teams can train agents on the kinds of tasks where the failures occur.

Get the detail on your own model

If you're building a model, we can take you through the mistakes behind its score. The public chart gives an overview; the write-up draws on a random sample of failed attempts to explain recurring behavior.

Each write-up covers:

  • The failure modes that dominate, including changes since earlier versions we've evaluated.

  • Where the model goes wrong, including failures that pass the conversation assessment.

  • How failures occur together or follow one another, described without exposing benchmark tasks.

  • Where training could help, informed by evidence from Toloka's own runs.

Write-ups go only to the team that built the model. Benchmark tasks and trajectories stay private.

How we measure

Each model gets five attempts at the same private enterprise tasks, with a simulated user taking part in each conversation. An attempt passes only if the final records are correct and the agent passes the assessment of required steps. The records are assessed automatically; a language-model judge assesses the steps. Tasks and grading are the same for every model.

How we analyzed the results

  • Failure categories: fixed rules map each graded check to a failure mode, and modes roll up into ten categories. One attempt can fall into several categories. Rates in the chart are shares of all attempts.

  • Silent failures: attempts that pass the conversation assessment but leave incorrect records, shown as a share of failed attempts.

  • Groups and medians: models are ranked by success on all five attempts (pass^5), using the October 28, 2026 leaderboard.

  • Repeated success: "every time" means success on all five evaluated attempts.

  • Statistics: rank correlation across models, bootstrap confidence intervals and Bonferroni-corrected tests for upgrades. The correlation between leaderboard performance and silent-failure share was +0.75. Differences of a few points may fall within statistical uncertainty.

  • Limitations: the tasks use simulated users, and automated grading can be wrong. A passing conversation assessment doesn't establish that a human would find the conversation flawless. Task coverage and repeated-success figures compare models; they don't predict how much an individual model will improve through training. The Argon and Opus comparison measures combined task coverage, not a tested system choosing between models. The upgrade analysis covers eight comparisons. Training results come from separate Toloka experiments, including a three-attempt measure for Qwen, and aren't directly equivalent to the leaderboard's five-attempt scores.

References

Find out where your model needs attention

Explore the Toloka Arena leaderboard to compare failure profiles, or connect with our team for a write-up on your own model. The enterprise RL Gyms behind the evaluation are also available to license for training and evaluation.

Subscribe to our newsletter

Product updates, case studies and the latest news from our team, straight to your inbox