Agentic Coding
How reliably AI agents solve terminal-based software engineering challenges, ranked by how often they succeed on all three attempts.
Domains
Models evaluated
Runs per task, all must pass
Agentic coding index
Composite pass^3 score: how likely a model is to succeed on all three attempts, averaged across tasks. Higher is better.
Error bars show approximate 95% confidence intervals; read close scores with caution.
Coding tasks available to license
Coding environments and evaluation data
License terminal tasks and containerised environments for training and evaluating your coding agents.
Request samples to see the format and content.
Methodology
Each benchmark has a published methodology covering its tasks, evaluation setup and scoring. Here's what goes into the results.
1
Tasks
Toloka Agentic Coding Tasks across engineering domains, from scientific computing to legacy systems.
2
Adapter
An adapter reads task’s directory and turns each task into Toloka Forge native objects.
3
Environment
Each trial gets its own isolated container with a shell, built from the task’s own setup. Trials run in parallel without sharing state.
4
Harness
Every model runs in Toloka Forge, our open-source agent harness, with the same loop, tools, prompts and graders. Token and time limits follow a per-model preset.
5
Runs
Every task runs three independent times.
6
Grading
Test inside the container results is a numeric reward. The adapter maps that onto our existing test-execution grading path, honouring the timeout each task declares for its own suite
7
Scoring
Models are ranked by pass^3, with approximate 95% confidence intervals. Cost per task covers all three runs, at public list prices.
8
Integrity
All tasks stay hidden, so there is nothing to train or tune on.
![](data:image/svg+xml,<svg display="block" role="presentation" viewBox="0 0 48 48" xmlns="http://www.w3.org/2000/svg"><path d="M 12.5 15.5 L 0 15.5 L 0 12.5 L 12.5 12.5 L 12.5 0 L 15.5 0 L 15.5 12.5 L 28 12.5 L 28 15.5 L 15.5 15.5 L 15.5 28 L 12.5 28 Z" fill="rgb(0, 0, 0)" height="28px" id="v_HNFUCmW" transform="translate(10 10)" width="28px"/></svg>)
What does Toloka Agentic Coding Tasks measure?
It evaluates AI agents on terminal-based software engineering challenges. Each task is a self-contained, Harbor-compatible environment with agent instructions, hidden automated tests and a verified reference solution. On Arena, every model runs in Toloka Forge, our open-source agent harness, and pass^3 requires success on all three attempts.
How is each model run?
Every model runs in Toloka Forge with the same loop, tools, prompts and graders, so score differences come from the model, not from a vendor's CLI or scaffolding. Toloka Agentic Coding Tasks run as-is. An adapter reads each task's manifest, Dockerfile, instruction and tests, and turns them into the engine's native task, prompt, tools, environment and grading rules. Each trial gets its own namespaced container stack, so trials run in parallel without sharing state.
How is a trial graded?
After the agent finishes, each task’s own hidden tests score the trial. A trial passes when its score reaches 0.5. Trials that never complete count as failures, and so do refusals.
How are failures analysed?
We trace every run and record how it ended. A run either finishes, hits its turn or time limit, runs out of token budget, is stopped by the repetition detector, produces an empty reasoning turn, or ends with an API error.
How are models scored?
Each task runs three times, and we rank by pass^3: how likely a model is to succeed on all three attempts. A model that solves every task 2 times out of 3 scores 67% on single attempts but 0% pass^3. We rank this way because a deployed coding agent has to succeed every time. Each score has an approximate 95% confidence interval. It's a normal approximation over the number of tasks, so it doesn't account for the three runs per task, and it shows no width at 0% or 100%. Read close scores with caution, whether or not their intervals overlap.
How is cost calculated?
Cost per task is the cost of all three runs of that task, averaged across tasks, at public list prices. Every input token is charged at the full rate (no cache discount), so providers compare fairly. It covers the agent's own tokens and excludes reasoning tokens that providers report separately.
How do you keep results honest?
Tasks are never published, so there is no split to train or tune on. A published set is never patched: defects are footnoted, and the next version is rebuilt in full. Every run is traced, and regressions are published along with gains.
Other benchmarks
RL Gyms and evaluation data
License tasks and environments for training and evaluating your models. Request samples to see the format and content.