← Blog

/

Arena

Arena

Same model, same tasks, 4.4× the cost: the harness nobody reports

Toloka Arena is live. See how your model ranks.

Every agent benchmark score comes out of a harness: the software that gives a model a task, runs its commands, decides when it’s done and tracks what it spent. Papers and leaderboards almost never say which harness they used.

That gap matters more than most people assume. We ran Haiku 5.5 on the same 50 Terminal-Bench tasks under two harnesses. Under Toloka Forge, it cost $20 and passed 71% of trials. Under Harbor, it cost $82 and scored 13 points lower. Almost half its trials ran to the 250-turn cap. The model, the tasks and the provider were identical.

To see how far this holds, we ran ten models through 3,000 trials on both harnesses: Harbor with its native Terminus-2 agent, and Toloka Forge, our open-source harness (Apache 2.0). Everything else was held constant: provider (with fallbacks off), temperature, token ceiling, reasoning effort, and a per-trial time limit matched trial for trial.

What we found:

  • Accuracy: On seven of ten models, the harness made no measurable difference. On three, it moved scores by 13 to 18 points, in both directions.

  • Cost: The harness changed the bill on every model, by up to 4.4×.

  • Run time: Harbor trials took longer on all ten models. The full board needed 2.7× more machine time.

  • Visibility: Some failures, like runaway loops and under-reported costs, only show up if you go looking for them.

Why Terminal-Bench: It doesn’t grade text. The model works in a real sandboxed terminal on a broken repository, and the tests either pass or fail. Our 50 tasks are Terminal-Bench-format environments built by Toloka and never published, so none of these models could have trained on them.

Most of the time, the harness is invisible

Accuracy difference per model, with 95% confidence intervals. Seven of the ten intervals cross zero: on those models the harness choice is not measurable at this sample size. Three do not, and they are 13 to 18 points wide.

This is the first thing to say, because it is the thing most likely to be assumed away in either direction. Running 150 trials of a model gives an accuracy figure with about ±6.4 points of play in it. Four of our ten models came out exactly 2.7 points apart between harnesses — four trials in 150 — and that is noise. So are the two exact ties, and so is Gemini's eight points.

Seven of ten is the honest headline: swap the harness under most models and the score does not move in any way you could defend.

Then there are the other three. DeepSeek v4.1 Flash runs 18 points better under Harbor. Haiku 5.5 runs 13 points worse, and Grok 4.7 13 points worse. These are not close calls — their intervals clear zero comfortably — and they run in both directions, which rules out the simple story that one harness is just better.

What makes those three different is the subject of the rest of this piece. It is not that the models are weaker or stronger. It is that each one trips a different part of the loop.

But it always shows up on the bill

Cost of the same 150 trials on each harness. Toloka Forge is cheaper on eight of ten models, by margins from 1.11× to 4.4×; Harbor is cheaper on Grok and GPT-6 Astra. Accuracy may be a coin flip — cost is not.

Where accuracy mostly refuses to separate the two, cost separates them on every single model. Toloka Forge costs less on eight of ten, from 1.13× on Fable up to 4.4× on DeepSeek. Harbor costs less on two — Grok 4.7 at 0.64× and GPT-6 Astra at 0.73× — and those are real wins, not noise; they are hundreds of dollars on a single arm.

Across the whole board we spent $1,731 on Harbor against $1,414 on Toloka Forge for identical work: 1.22× in aggregate, 1.32× at the median.

There is no sentence that begins "Harbor is cheaper because." The two arms where it wins get there by opposite routes. On Astra it sends more prompt and takes more turns, and still wins on price, because its prompt is more cache-stable across turns: a 91% cache hit rate against 84%, which leaves it paying full price on 1.6× fewer tokens. On Grok it wins by sending half as much prompt, with a worse hit rate. Both harnesses end up paying for the same uncached prompt within 2%.

The same model, on the same tasks, can cost 4.4× more depending only on which harness runs it. That is the number to take to a budget meeting.

A trial is mostly waiting — but not in the same place

Both harnesses record enough to split a trial into time spent waiting on the model and time spent inside the environment: starting containers, running commands, reading back output. That split separates a slow model from a slow harness, and almost nobody reports it.

Share of each trial spent outside the model. Toloka Forge spends 0–3% on every model; Harbor spends 10–22% on eight of ten, 36% on GPT-6 Sol, and 1% on DeepSeek, whose trials run so long the overhead disappears inside them.

Toloka Forge spends between 0% and 3% of a trial outside the model. Harbor spends between 10% and 22% on eight of the ten and 36% on GPT-6 Sol. The exception is DeepSeek at 1%, and only because its trials run so long that the same overhead vanishes inside them. That time is not generation. It is the harness.

Sol is the cleanest demonstration, because there Harbor is actually the faster of the two inside the model and still loses. It spends 77 seconds waiting on tokens where Toloka Forge spends 97 — and still finishes later, 125 seconds against 102, because more than a third of its trial happens somewhere else entirely.

Harbor's median trial is longer on every one of the ten models — from 1.08× on Gemini to 5.45× on Sonnet. That is worth knowing before you size a run, because it is the difference between an eval finishing overnight and not.

The harness does not change how fast a model writes

The obvious explanation for longer trials is that the model generates more slowly under one harness. It doesn't.

Milliseconds per token against response size, within each model. Harbor is slower on Opus, Astra and Grok, faster on Sol and DeepSeek, level on the rest — the spread tracks the model, not the harness.

Comparing per-token speed only works size-matched — a harness that writes longer answers looks faster per token for free — and it only works within a model, because the two harnesses do not have the same mix of models in their call pool. Done properly, there is no systematic harness effect. Harbor runs between 0.67× and 1.71× the engine's milliseconds-per-token, and the direction flips model by model: slower on Opus, Astra and Grok, faster on Sol and DeepSeek, level on Gemini, Fable, Sonnet and Haiku. It varies by model, not by harness.

So the longer trials are not slower tokens. They are more turns, more tool calls, and more time between them.

The cost is in the turns

Median turns and tool calls per trial. Harbor takes more turns on nine of ten models and up to 4.5× the tool calls.

Cost in an agent loop is turns multiplied by the size of each turn, and the turn count is where the two diverge. Harbor takes more turns on nine of ten models — 40 against 14 on Haiku, 29 against 15 on Luna — and more tool calls almost everywhere, up to 108 against 24 on DeepSeek.

Grok is the exception that proves it: 24 turns against 26, 24 tool calls against 26, near-identical loops. It is also the arm where Harbor's cost advantage is largest. When the loops match, the cache behaviour decides the bill.

Some turns buy nothing

A healthy agent turn ends in an action. If it doesn't — if the model reasons, produces no command, and hands back — the turn is paid for and the trial has not moved. Count turns against tool calls and that shows up directly.

Turns that produced no action, as a share of turns. Under Harbor this spans 182 points across the ten models; under Toloka Forge it stays inside a 12-point band.

Under Toloka Forge the figure sits between −2% and +10% on every model — a twelve-point band. Under Harbor it ranges from −121% to +61%, and on three models it goes positive: Haiku at +61%, Sonnet at +25% and Gemini at +14%, meaning turns outnumber actions and the loop is spending money to stand still.

That is the whole of Haiku's 4.21× cost. The two harnesses agree almost exactly until the loop starts: Harbor's uncapped Haiku trials run a median of 14 turns, and Toloka Forge runs 14. But 70 of 150 Harbor trials never finish — they run to the 250-turn cap, and inside those trials 74% of turns produce no command at all.

What the loop does is not in doubt. What triggers it is a narrower question, and we tested the obvious answer and ruled it out. Harbor asks for actions as JSON inside free text, where Toloka Forge uses native function calling that makes a malformed action impossible — so we swapped Terminus-2's parser for the stricter one and re-ran. The trials still capped, and parse rejections went up rather than down. The action protocol is not the cause.

What remains is a specific interaction between this model and this loop. We can measure it exactly and we have not isolated it, which is why Haiku's 13-point deficit belongs to the pair rather than to Harbor on its own.

Sonnet is the same shape in miniature and worth watching for that reason. Two trials out of 150 hit the cap — and those two are 27% of that arm's $277 bill. A loop that does this rarely still sets the price.

How each harness fails, and how loudly

How trials end, by share. Toloka Forge stops trials early on budget and stuck detection; Harbor runs them to an error or to the turn cap.

On seven of the ten models both harnesses complete essentially every trial. Where they diverge, they diverge by design.

Toloka Forge stops trials early. It ends trials when they exhaust a token budget (19% of Gemini) and when it detects the agent is stuck (17% of DeepSeek). These stops are cheap, and they are also surrender: on Gemini it gives up on roughly one trial in five. Whether that costs accuracy we cannot say from this data — the gap there is inside the noise.

Harbor runs to the wall. It caps 47% of Haiku trials, and ends 44% of DeepSeek trials in an error state — which turns out to understate what happened. Every one of those 66 trials was still working when the matched wall clock cut it off: a median of 9,027 seconds against a 9,000-second limit, all 66 at the wall. And 39 of them had already passed. Harbor was stopped mid-work on nearly half this arm and still finished 18 points ahead, which makes that +18 a floor rather than a result.

Harbor was precise about what happened, for the record: it named every one of those 66 terminations AgentTimeoutError. It was our own bundle format that flattened that into a generic error state, and we say so because we nearly drew the wrong conclusion from our own data.

The ending that genuinely is hard to see is the turn cap. All 74 Harbor trials that hit the turn limit returned without raising anything at all, which makes their outcome indistinguishable from a trial that solved the task on turn nine. Nothing fails; you find them by comparing the turn count against the configured cap. If you were monitoring a long run by outcome status — the obvious thing to do — you would not see the runaway. Toloka Forge surfaces the turn limit as a named reason, alongside token budget exhausted, agent stuck, provider refusal and finished, so the same event shows up without being looked for.

The same holds for cost. On the seven models where both harnesses self-report a bill, Toloka Forge matches what was actually billed to the cent on all seven. Harbor matches on six and under-reports by 4.6× on DeepSeek — $16 claimed against $74 paid, because that model returns most of its output as reasoning tokens and Harbor's accounting prices them at zero. One arm in seven is a low rate. It is also the arm you would least want to be wrong about, and you would only catch it by reconciling against the invoice.

What this means if you're choosing one

Harbor is better at finishing. It does not abandon trials. Where a model is slow but still converging, Harbor sees it through, and on DeepSeek that is worth 18 points of accuracy. Its prompt is more cache-stable on some models, which can make it cheaper on exactly the expensive frontier models where cost matters most. It also instruments the model-versus-environment split natively, which is what made this comparison possible at all.

Toloka Forge is better at finishing quickly, predictably, and cheaply in machine time. It completes a trial faster on every model measured, between 1.08× and 5.45×, and spends almost nothing outside the model. Running this whole board took 193 trial-hours on Toloka Forge against 523 on Harbor — 2.70×, or 330 extra hours of occupied runners for identical work. If your constraint is container capacity, CI minutes or how long a board takes to come back, that is the ratio that binds, and it is more than twice the gap in the token bill. It costs less on eight of ten. Its loop behaviour barely varies across models — a twelve-point band against Harbor's 182 — so a run that is sized correctly for one model is sized correctly for the next. It detects stuck agents and exhausted budgets and says so in a named reason. And its cost accounting has been exact on every model we have run.

The trade is symmetric, and it is not mainly about accuracy. On seven of ten models the accuracy question does not have an answer at this sample size. The questions that do have answers are what it costs, how long it takes, and whether you will be able to tell what happened afterwards.

Takeaway

A benchmark score without its harness is incomplete. On seven of our ten models, swapping the harness made no measurable difference to accuracy. On the other three, it moved scores by 13 to 18 points, in both directions. You can’t tell which case you’re in without measuring.

Cost and time changed on every model. Harbor cost 1.22× more in API spend and needed 2.7× more machine time: 523 trial-hours against 193. These are separate budgets. If you plan only for tokens, you’ll still run short of compute.

None of these differences came from the model. They came from the loop around it: more turns, more tool calls, up to a third of each trial spent outside the model, and turns that produced no action at all.

Before you compare two agent results, check three things:

1. Which harness produced each number. If you can’t name it, you’re not comparing models.

2. Cost per solved task, not just cost per trial. A harness that stops trials early can look cheaper per trial.

3. How trials ended. Harbor reported 74 trials that ran into the turn cap as normal finishes, and under-reported DeepSeek’s cost by 4.6×. Neither shows up in pass/fail status. You find them by comparing turn counts against the cap and checking costs against the invoice.

Run Toloka Forge yourself. It is open source under Apache 2.0 on GitHub.

Test your agents on tasks they’ve never seen: Every environment in this study was built by Toloka and never published. License them off the shelf browse datasets or have us build custom ones talk to us.

Subscribe to our newsletter

Product updates, case studies and the latest news from our team, straight to your inbox