Eenterprise tool use
Evaluates how reliably AI agents complete realistic enterprise workflows across 11 business domains.
Industries
Models evaluated
Runs per task, all must pass
Agentic intelligence index
Composite pass^5 score: how likely a model is to succeed on all five attempts, averaged across tasks. Higher is better.
Error bars show approximate 95% confidence intervals; read close scores with caution.
Off-the-shelf RL Gyms and expert data
License tasks and environments for training and evaluating your models. Request samples to see the format and content.
Methodology
1
Tasks
Realistic enterprise workflows from 11 industries.
2
Setup
The agent works in a sealed sandbox with the company’s databases, tools and policy. A simulated customer with a hidden goal talks to it until the request is resolved.
3
Harness
Every model runs on tolokaforge, Toloka’s open-source harness, and is called and graded the same way.
4
Runs
Every task runs five independent times.
5
Grading
Not just pass/fail: the final records must match the reference solution, and a judge checks every step against task-specific rubric criteria.
6
Scoring
Models are ranked by pass^5, with 95% confidence intervals. Cost per task is shown at public list prices.
7
Integrity
All tasks stay hidden, so there is nothing to train or tune on.
![](data:image/svg+xml,<svg display="block" role="presentation" viewBox="0 0 48 48" xmlns="http://www.w3.org/2000/svg"><path d="M 12.5 15.5 L 0 15.5 L 0 12.5 L 12.5 12.5 L 12.5 0 L 15.5 0 L 15.5 12.5 L 28 12.5 L 28 15.5 L 15.5 15.5 L 15.5 28 L 12.5 28 Z" fill="rgb(0, 0, 0)" height="28px" id="v_HNFUCmW" transform="translate(10 10)" width="28px"/></svg>)
What does Enterprise tool use measure?
Whether an AI agent can complete enterprise workflows reliably, not just once. Pass^5 measures agents’ ability to navigate business systems, follow company policies, and resolve customer and employee requests across five attempts in a simulated business environment.
How is each model run?
Every model runs on tolokaforge, Toloka’s open-source harness for agent evaluation. It puts the agent, a simulated customer and the company’s tools in one sandbox, calls every model through the same interface and grades every trial the same way. Because the harness is public, anyone can inspect how models are called and how trials are graded.
The agent starts in a sealed sandbox with no internet, with the company’s databases, its tools (CRM, ERP, ticketing) and its policy. A simulated customer with a hidden goal talks to the agent until the request is resolved or the turn limit is reached. The agent must stay within policy, including refusing or escalating when required. Every message and tool call is recorded and passed to grading.
Tasks, prompts and graders never change per model.
How is a trial graded?
A trial passes only if it gets both the outcome and the process right.
Outcome: the final database state must exactly match the state produced by the reference solution.
Process: a judge reviews the transcript, tool calls and database changes against task-specific rubric criteria, and never sees the reference solution.
Trials lost to infrastructure errors are excluded. Refusals count as failures.
How are failures analysed?
We trace every run and classify failures, including incorrect results, wrong tool calls, policy violations and turn limits.
How are models scored?
Each task runs five times, and we rank by pass^5: how likely a model is to succeed on all five attempts. A model that solves every task 4 times out of 5 scores 80% pass@1 but only 33% pass^5. We rank this way because a deployed agent has to succeed every time.
Every score has a 95% confidence interval. Models whose intervals overlap are not clearly ranked.
How is cost calculated?
Cost is the average per task at public list prices, with every input token at the full rate (no cache discount), so providers compare fairly. It covers the agent’s own tokens, not the simulator or judge, and excludes reasoning tokens that providers report separately.
How do you keep results honest?
Tasks are never published, so there is no split to train or tune on. A published set is never patched: defects are footnoted, and the next version is rebuilt in full. Every run is traced, and regressions are published along with gains.
Other benchmarks
Off-the-shelf RL Gyms and expert data
License tasks and environments for training and evaluating your models. Request samples to see the format and content.