GPT-5.6 got smarter. Then it kept acting.

Sol and Terra beat GPT-5.5 on most public agent evaluations. On 711 identical private enterprise workflows, every matched GPT-5.6 configuration scored lower. The trajectories show why: OpenAI’s newest GPT-5.6 variants more often reach the right context and then keep acting — duplicate records, re-attempted values, extra state — even as it checks more in aggregate and avoids a large class of airline tool errors. In the workflows where a reliable GPT-5.5 becomes a reliably failing GPT-5.6, two-thirds of the failures are surplus action or a wrong written value; retrieval failures are almost absent.

Executive summary

• The result. On 711 frozen enterprise workflow tasks, graded only on the exact final database state, every effort-matched GPT-5.6 configuration scores below GPT-5.5: Sol -3.7 and -4.0 points, Terra -8.7 and -6.7 (all Holm p <= 0.0016). For scale, the GPT-5.4 to GPT-5.5 upgrade was worth +3.0 points on the same tasks — Sol hands back one upgrade step, Terra medium nearly three.
  
  

• The same models lead in public. Sol beats GPT-5.5 on seven of eight deduplicated public agent-evaluation lanes, Terra on six of seven. Both readings are correct: our panel tracks the public consensus across seventeen other models at Spearman 0.90, and cross-vendor scatter there is ±9 points — a 4-to-9-point within-family regression is invisible at that resolution and only a paired same-task design resolves it.
  
  

• What actually breaks. We reviewed every strongly flipped workflow, all 153. Two-thirds of the regressions are surplus action (a duplicate case, ticket, or notification, or the correct action followed by more writes) or a wrong written value; retrieval failures are almost absent. GPT-5.6 typically does the intelligent part, then keeps acting.
  
  

• Why to believe it.The direction survives six outcome definitions, deletion of any domain or task family, and every grading-artifact exclusion — including the five artifact candidates our own audits went looking for and found (worst-case shift 0.2 points). Eleven of eleven other within-family upgrades move up on this panel. The one alternative we cannot exclude is hidden serving-side drift between collection windows, and we say so.
  
  

• What to do. Do not upgrade production agents on leaderboard evidence; re-run your own workflows paired, gate on terminal-state checks rather than read-backs, treat reasoning effort as a model-specific setting (xhigh helps Terra, not Sol), and instrument duplicates, value reversals, and extra records as leading indicators.

GPT-5.6 is a real upgrade.

At their reported peak settings, Sol scores above GPT-5.5 on seven of eight deduplicated public agent-evaluation lanes in our July 15 ledger. Terra scores above it on six of seven lanes with a published Terra result. The family leads on coding, terminal use, IT operations, research work, and automation.

Then we put it inside enterprise systems.

We ran four GPT-5.6 configurations across a complete seven-domain suite: 16,680 GPT-5.6 trajectories in all. The confirmatory comparison uses the 711-task six-domain intersection across GPT-5.5 and GPT-5.6, or 21,330 focal trajectories. The agents had to inspect records, read policy, choose an action, call tools, and leave the database in an exact expert-authored state. Every task ran five times. The final state, not the prose, determined success.

On the 711 tasks shared with GPT-5.5, all four matched comparisons went backward:

• GPT-5.5 medium -> Sol medium: -3.7 percentage points.
• GPT-5.5 medium -> Terra medium: -8.7 points.
• GPT-5.5 xhigh -> Sol xhigh: -4.0 points.
• GPT-5.5 xhigh -> Terra xhigh: -6.7 points.

That is the surprising result. For scale: on the same 711 tasks, the GPT-5.4 to GPT-5.5 upgrade was worth +3.0 points. Sol gives back roughly one such step; Terra medium gives back nearly three.

The more important result is what happened underneath it.

GPT-5.6 did not simply stop checking. It performed 3.2 to 5.8 points more target-specific read-back after its final write — a rise that is broad-based for Terra but concentrated in one domain for Sol, a distinction we quantify below. Aggregate tool-error incidence fell by 11.7 to 16.6 points, almost entirely because it avoided a large class of airline schema validation failures; other domains were much smaller and sometimes moved the other way.

Lower workflow success coincides with a separation between activity and terminal-state control.

Across the four matched upgrades, GPT-5.6 made 4.8 to 8.5 points more exact duplicate calls, 1.8 to 3.8 points more attempted write-value reversals, 7.8 to 12.0 points more read-write continuation, and 2.6 to 4.3 points more graded extra records.

The model often did the intelligent part. Then it kept acting

Figure 1. Public rows preserve their native units and are not pooled. They compare GPT-5.6 max with GPT-5.5 xhigh. Our private rows compare effort-matched medium and xhigh settings on identical tasks. The point is not a universal ranking; it is evaluation-contract dependence.

The claim, precisely

This article makes three claims.

First, we observe a paired benchmark regression under matched visible configurations. The tasks, policies, grading configuration, enabled tools, user simulator, provider preset, temperature, token limits, and response/schema settings match. GPT-5.5 and GPT-5.6 were collected in different windows, so we cannot identify equality of hidden provider snapshots or live backend implementations. This is not a laboratory decomposition of model weights from every runtime factor.

Second, the ordering persists across exact pass rate, repeated reliability, absence of extra state, a policy-sensitive and completeness diff score, and a fixed weighted failure utility. It remains negative when any one domain is removed, although not every alternative outcome interval excludes zero.

Third, the trajectories localize the behavioral change. The evidence is descriptive, not causal mediation: GPT-5.6 combines more aggregate target read-back and a domain-concentrated reduction in tool errors with more duplicate calls, value reversals, extra records, and read-write continuation. A purposive, identity- and grade-visible source-turn audit localizes policy application and mutation as common early scored divergences among selected high-contrast cases.

That combination is what we call a state-control regression: the observed GPT-5.6 trajectories contain more target read-back in aggregate and, in airlines, fewer rejected tool calls, while exact terminal-state control worsens.

Figure 2. Equal-domain paired task-cell differences. Read-back is a structured event, not proof that the returned state was evaluated correctly, and its rise is logistics-concentrated for Sol (quantified in the event section). “Write after intervening read” is a separate action-path marker and is not conditioned on the final-write read-back.

GPT-5.6 checks the record more often — and still does not leave it alone.

Nine alternatives, one operational pattern

The task panel, policies, tools, and grader are fixed by construction. Their equality is part of the comparison contract, not a hypothesis we need the data to discover. The analysis therefore starts with the explanations that can still account for a lower score on that fixed panel.

We organized nine remaining alternatives into an explanation ledger. The rows are not mutually exclusive: runtime drift, task difficulty, retrieval, policy application, and state control can coexist.

Figure 3. Every row is tied to a machine-readable analysis. “Supported” means the observed behavior is consistent across the relevant diagnostics; it does not mean the row is a causal mediation estimate.

Grading artifacts fail as a complete explanation — and we went looking for them twice. The source-turn audit exposed two QSR tasks where notification insertion order changed the state hash despite behaviorally identical recipient/payload sets; removing both shifts any primary estimate by about 0.13 points. The exhaustive flip review described below surfaced three more candidates that we disclose and quantify in that section: a QSR task graded on a timestamp that differs only by a missing timezone suffix, a second order-sensitive notification task, and a manufacturing task with an internally inconsistent grade record. No combination of these exclusions moves any primary estimate by more than 0.2 points. The regression remains.

One explanation cannot be eliminated. Visible runtime signatures match, but the collection windows differ: GPT-5.5 was collected in April and May, GPT-5.6 in July. We cannot prove that an unobserved provider snapshot or live service was identical. That is the main identification boundary.

The observed data also make a pure floor effect, a panel-wide increase in tool errors, and a general absence of checking implausible as complete explanations. The losses are not concentrated at the floor or ceiling. Aggregate tool errors fall because of airlines. Target-specific read-back rises in aggregate and falls in no domain — although, as the event section details, the size of the rise is itself domain-concentrated for Sol.

Retrieval matters in individual cases, but it is not the modal explanation in the selected audit. Policy application and mutation are.

Within the selected high-contrast cases and structured event ledger, the patterns most consistent with the trajectories sit later: translating a rule into the correct state, controlling subsequent mutations, and stopping at the target state. Hidden cross-window runtime drift remains unresolved.

So what: the regression is not explained by the grader, impossible tasks, more rejected tool calls, or less checking. The evidence points downstream, where a model turns information into state and decides whether to act again.

The ordering persists across several workflow outcomes

The primary estimand is deliberately simple. For each task and model, we average the five binary outcomes into a success probability from 0 to 1. We subtract GPT-5.5 from GPT-5.6 within the same task, average within each domain, and give the six domains equal weight.

This preserves the repeated-trial cell and stops large domains from dominating the result.

The 95% intervals come from a task bootstrap stratified by domain. Paired task-label randomization tests the task-superpopulation question, with Holm correction over the eight frozen primary contrasts. We separately report exact sign randomization over the six domain effects. Those are different uncertainty targets and should not be collapsed into one p-value.

Figure 4. Panel A contrasts task and task-family cluster uncertainty. Panel B changes the outcome definition. Panel C changes the reliability threshold. Panel D separates task-level evidence from six-domain portability. A fixed panel, a population of similar tasks, and a population of domains are not the same estimand.

The four exact-pass effects are:

In absolute terms, domain-balanced exact success is 65.7% for GPT-5.5 medium, 62.0% for Sol medium, and 57.0% for Terra medium. At xhigh it is 66.2%, 62.2%, and 59.5%, respectively.

Drop any one domain and all four remain negative. The leave-one-domain ranges are -5.5 to -2.4 for Sol medium, -10.1 to -7.5 for Terra medium, -4.7 to -3.3 for Sol xhigh, and -8.1 to -5.9 for Terra xhigh. Task-level uncertainty supports all four effects on this frozen domain panel. Domain-level inference is necessarily limited with six domains: exact domain-sign p-values range from 0.031 to 0.188.

The result also survives dependence among templated tasks. An outcome-blind rule groups the panel into 86 domain-specific task families from task identifiers.

Resampling whole families leaves all four generation intervals below zero: [-5.6, -1.9], [-10.2, -7.1], [-6.0, -2.1], and [-8.6, -4.8] points. Deleting any one family changes a headline estimate by at most 0.44 points.

The ordering also survives outcome changes:

• If we ignore omissions and wrong values and ask only whether the model avoided extra state, GPT-5.6 still loses 2.6 to 4.3 points. (This is the same graded measurement that appears as “extra records” in the event ledger below; we count it once as an outcome, not twice as independent evidence.)

• Under a policy-sensitive and completeness diff score, which penalizes any missing record plus lifecycle, approval, routing, priority, and type errors, it loses 1.5 to 5.7 points.

• Under a fixed weighted failure utility, loss worsens by 2.1 to 5.3 points.

• At the stricter repeated-reliability thresholds of at least 4/5 and exactly 5/5, all four generation comparisons remain negative.

Exact pass, no-extra-state performance, and weighted failure loss support the regression across all four generation contrasts. The policy-sensitive/completeness score supports three of four after correction. Strict 5/5 reliability remains directionally lower but is not uniformly distinguishable from zero.

So what: this is not a quirk of one pass threshold, one large domain, one task template, or one exact-match utility. Teams changing the acceptance metric would still face the same migration decision.

The losses are not on impossible tasks

One hypothesis was that GPT-5.6 looks worse because this panel is simply too hard. That predicts the biggest gaps at the bottom of the difficulty curve.

We estimated task difficulty without using any focal model. Within each domain, we averaged success across four context models and divided tasks into five equal-sized difficulty bins.

The result is almost the opposite of a floor effect.

Figure 5. Difficulty is estimated from GPT-5.4, GPT-5.4 xhigh, GPT-5.5 high, and Sonnet 5. Focal GPT-5.5 and GPT-5.6 configurations are excluded. Bands are task-bootstrap intervals within domain and quintile.

For Sol medium, the paired change is essentially zero in the hardest quintile (+0.1 points) and small at the ceiling (-1.3 points). The losses peak in the middle-high region: -7.9 points and -5.4 points, both with intervals below zero.

Terra shows the same shape at larger magnitude, including -10.2, -11.3, and -13.2 points across the central bins.

These are not hopeless tasks where every model flails. They are workflows the wider panel often solves. That matters because it points away from missing base capability and toward how capability is converted into controlled action.

So what: GPT-5.6 loses most where it already has enough capability to succeed. More knowledge or raw problem-solving power is therefore unlikely to be the whole remedy; execution control is the higher-leverage target.

The model checks more, then writes again

To understand the trajectories without loading the whole corpus into memory, we built a grade-blind streaming event parser. It reduced 21,330 trajectories across the six focal configurations into semantic workflow events with source file and turn pointers.

The parser tracks:

• reads before the first mutation;

• grounded entity breadth;

• mutations and distinct mutated entities;

• target-specific read-back after a final write;

• exact duplicate calls;

• write-value reversals on the same target and field;

• writes after an intervening read;

• tool errors, retries, and recovery;

• completion claims without a successful mutation;

• graded extra records from the deterministic state diff.

We ran a 200-row structural and privacy validation after fixing an overbroad private-identifier check. All 200 passed call/result linkage, source-pointer, JSON-roundtrip, and privacy checks. This does not validate every event label semantically. The extractor favors precision over recall: a broad list call without an exact target identifier does not receive target-read-back credit.

Figure 6. Equal-domain paired task-cell changes. Each cell reports its estimate and BH q-value across the declared 84 diagnostic feature-by-contrast tests. Conditional rates use explicit mutation, error, or claim opportunities. These are behavioral diagnostics, not mediation estimates.

The top panel rules out a simplistic “more tools, more failures” story. Workflow volume moves modestly. Some configurations read slightly more before the first mutation; others slightly less. Mutation count is nearly flat.

The bottom panel is where the behavior changes. Across all four generation contrasts, the increases in target read-back, duplicate calls, attempted value reversals, read-write continuation, and graded extra records survive the full event-family correction (q <= 0.0013).

GPT-5.6 reads back more, mostly in one place

After the final mutation, GPT-5.6 is, in the equal-domain aggregate, 3.2 to 5.8 points more likely to perform a target-specific read-back. Two qualifications keep that number honest. First, a read event alone does not show that the returned state was interpreted correctly. Second, the rise is not equally broad-based: Terra’s read-back increase persists when any single domain is removed, but Sol’s is concentrated in logistics. Excluding that one domain, Sol medium’s rise shrinks from +3.6 to +0.7 points, and Sol xhigh’s becomes indistinguishable from zero.

Read-back also falls in no domain for any configuration. The honest summary: checking did not decline anywhere, rose sharply in logistics, and rose panel-wide only for Terra — which is still enough to rule out “GPT-5.6 simply stopped checking” as the explanation for the regression, and is why we build nothing stronger on it.

GPT-5.6 also continues after intervening reads

The share of mutation-opportunity trials containing a write after an intervening read rises by 7.8 to 12.0 points. This statistic does not condition on the final-write read-back above, and it does not show that verification caused the later action. It shows more read-write continuation in the action path.

GPT-5.6 repeats and reverses more

Exact duplicate calls rise by 4.8 to 8.5 points. Attempted write-value reversals — a later write that returns a field to a value the model itself wrote earlier in the same episode (A→B→A) — rise by 1.8 to 3.8 points.

A duplicate can be intentional, so it is not itself an error label. The grader’s deterministic state diff separately shows 2.6 to 4.3 points more extra records; that is a graded outcome joined to the ledger rather than a trajectory event, so we treat the path markers (duplicates, reversals, continuation) as the grade-blind evidence and the extra-record outcome as where the grader agrees with them. Unlike the read-back rise, these increases are broad-based: dropping any single domain leaves them positive.

One airline regime drives the tool-error decrease

Tool errors fall by 11.7 to 16.6 points in the equal-domain aggregate, but the decline is almost entirely concentrated in airlines. There, GPT-5.5 frequently violated Zendesk call schemas and GPT-5.6 adhered to them. The sampled task tool schemas are identical, so this is consistent with better airline schema adherence, not a more forgiving sampled interface. Outside airlines, the equal-domain changes range from +0.0 to +5.7 points: no general tool-error improvement appears in the other five workflows.

That makes the overall result more interesting. Increased tool rejection is not a panel-wide explanation for the lower score. In the airline schema path, GPT-5.6 gets more calls accepted while the aggregate workflow ordering still favors GPT-5.5.

So what: the upgrade improved access to the action channel without improving terminal-state discipline. Better schema adherence and more read-back are useful, but neither is a reliable proxy for completing the workflow correctly.

What is wrong in the final state?

The event ledger describes the path. The grader diff describes what remained broken.

We assign every failed trial to a mutually exhaustive state-diff anatomy:

1. extra side effect;

2. missing action or record;

3. lifecycle or approval;

4. routing, priority, or type;

5. quantitative, date, or value;

6. other or unlocalized.

If a trial spans multiple categories, it contributes equal fractional mass across them, so each failed trial still sums to one.

Figure 7. Left: total failure probability by category. Right: each category’s descriptive contribution to the change in failure probability. The decomposition is exhaustive but post-outcome; it is not causal mediation.

Extra side effects rise in all four GPT-5.5-to-GPT-5.6 comparisons and in both medium-to-xhigh effort comparisons. Under the prespecified descriptive attribution, extra-side-effect mass accounts for 2.27 points of Sol medium’s 3.66-point increase in failure probability and 2.24 points of Terra medium’s 8.68-point increase.

The same domain-concentration standard we applied to tool errors applies here. The extra-side-effect rise is largest in travel-marketplace support and slightly negative in manufacturing; deleting the travel domain cuts Sol medium’s extra-side-effect contribution from +2.27 to +1.01 points. The direction survives every leave-one-domain deletion in all four contrasts, so the finding is panel-robust in sign — but its magnitude leans on the domains where agents create tickets and cases, which is also where production deployments would feel it.

The remaining difference is model-specific. Terra adds a large routing/priority/type component: +2.74 points versus +1.10 for Sol. Sol partly offsets its losses with fewer quantitative/date/value errors.

The largest recurring loci are not abstract reasoning failures. They are operational objects:

• extra Dynamics cases;

• extra Zendesk tickets;

• extra notification records;

• wrong case type;

• wrong ticket priority.

That is what state-control regression looks like in production: not necessarily a nonsensical answer, but one extra case, one overwritten field, one priority that changed after the correct decision, or one workflow that stopped a lifecycle transition too early.

So what: the regression lives in externally consequential state, not merely in answer style. These are precisely the errors that plausible prose and tool-call acceptance fail to reveal.

The transcript audit: the policy was often found

Automated event definitions cannot tell us whether two superficially similar reads reflect the same reasoning. So domain auditors reviewed 240 selected GPT-5.5/GPT-5.6 trajectory pairs across the six domains, balanced between strong fixes and strong regressions.

The audit was intentionally diagnostic. Model identity and grades were visible, and cases were selected because outcomes changed. It cannot estimate population prevalence or provide independent mechanism validation. Its purpose is to ask whether the source turns are consistent with the proposed interpretation.

Sixty pairs received a second label. Exact overall-dominance agreement was 95.0%, with Cohen’s kappa 0.903.

This agreement does not validate the stage labels. The primary strict judgment aligned with the grade direction in 228 of 230 scorable cases, partly reflecting the transition-selected design. The audit also found the two notification-order artifacts discussed earlier.

Figure 8. Left: audited stage scores move with the selected grade transitions. Right: first differing rubric stage in the 121 GPT-5.5-pass/GPT-5.6-fail cases. Use these counts for interpretation, not prevalence.

Among the 121 selected regression cases, the first differing stage under the fixed conceptual rubric was:

• policy application in 69 cases;

• mutation in 26;

• exploration in 23;

• retrieval in 1;

• no scored divergence in 2.

This is not evidence that retrieval is universally solved. It is evidence that, in these high-contrast cases, the model often reached the relevant context and diverged when converting that context into an operational decision.

So what: adding retrieval, a longer context window, or another policy-search step will not fix the selected failures by itself. The system must validate the decision and the resulting state after retrieval succeeds.

Other manifestations include a duplicated payroll notification and a refund that included a nonrefundable component. There are fixes too: GPT-5.6 correctly selected a nuanced QSR subtype that GPT-5.5 missed, and in another travel case it avoided a duplicate escalation. The new family is not uniformly less controlled. It has a different failure surface.

Every strong flip, reviewed in two model-assisted passes

The audit above interprets a selected sample. For the flips that matter most to a migration decision, we went further: every task cell that strongly flipped between generations — solved on at least four of five runs by GPT-5.5 and at most one of five by GPT-5.6, or the reverse — was reviewed end to end. That is 123 strong regressions and 30 strong fixes across the four matched contrasts. Nothing in this section is sampled. Each case was read independently twice by a reviewer model working from the task definition, both full trajectories, and the grader’s state diff, and a separate skeptic re-examined a fixed quarter of the regression reviews in each pass: of 61 skeptic checks, 58 confirmed the account, 3 adjusted only the mechanism label, and none was refuted.

Grouping the mechanism labels into stable families (both review passes land within a few cases of each other), the 123 strong regressions break down as:

Roughly two-thirds of the strong regressions are the model doing more than asked, or writing the wrong value — not failing to find information. Retrieval is nearly absent, exactly as in the human audit (1 of 121). The 30 strong fixes are the mirror image: GPT-5.6 most often wins by applying a policy correctly where GPT-5.5 rushed (8 cases), by finishing work GPT-5.5 left incomplete (7), and by writing the right value where GPT-5.5 did not (6) — including one case where it resisted an “executive override” that GPT-5.5 accepted in violation of an order-status rule.

Three of the flip stories compress the whole pattern:

What the reviewers were told to find wrong with us

Every reviewer also had a standing instruction to hunt for reasons to distrust the grade itself — ambiguous task wording, order- or format-sensitive state hashes, unequal simulator behavior. We would rather publish those ourselves than have a reader find them. Across both passes, 71 of the 153 cases attracted at least one such note (54 in both passes). Most concern policy wording that admits a defensible alternative reading — real, disclosed, and inherent to policy-driven work. The two findings with teeth were these.

First, the review surfaced two new grading artifacts, both in QSR and both hitting one contrast: a task graded on a timestamp that differs from the target only by a missing timezone suffix for the same instant, and a second notification task graded on insertion order with an identical notification set — the same sensitivity class as the two artifacts our human audit had already caught. It also flagged one manufacturing task whose failing grade records are internally inconsistent, which we treat as a harness defect. Excluding the two new tasks shifts any primary estimate by at most 0.19 points; excluding all five artifact candidates at once shifts any estimate by at most 0.11 points, and several regressions grow slightly. Every direction survives every exclusion.

Second, in about a fifth of the strong regressions the reviewers noted that the simulated user behaved differently across the paired runs — a repeated request in one dialogue but not the other, or an early stop. This matters less than it first appears, for a reason worth being precise about: the flip criterion is not one pair of runs but at least 4/5 against at most 1/5, and when we checked all five runs of the failing configuration in each such case, 21 of 22 failed with the same signature in at least four of five runs. The user simulator is itself a model responding to the agent; when GPT-5.6’s phrasing reliably elicits a repeat request and GPT-5.6 then reliably converts that repeat into a duplicate ticket, that is reproducible joint behavior of the model in the environment, not sampling luck. We disclose it as a property of the benchmark rather than pretend the stimulus is fixed beyond the first turn.

So what: an exhaustive, adversarially re-checked reading of every strong flip lands on the same diagnosis as the statistics: the regression lives in surplus action, wrong written values, and policy-to-state translation — not in retrieval, not in the grading artifacts we identified, and not in single-draw simulator luck. And the fixes show the same levers moving the other way, which is what a genuinely different failure surface looks like.

Xhigh changes the error mix differently in Sol and Terra

The within-family medium-to-xhigh contrast is our most direct observed configuration comparison for the “just let it think longer” response.

Sol xhigh adds 1,088 reasoning tokens per trial over Sol medium. Mean exact success changes by +0.2 points, with a 95% CI of [-1.2, +1.6]. It is statistically flat.

Terra xhigh adds a similar 1,066 tokens and gains +2.5 points, with a 95% CI of [+0.8, +4.3].

The difference between those two responses is itself statistically supported, not an artifact of comparing a significant result with a non-significant one: Sol’s effort response minus Terra’s is -2.3 points, with a 95% CI of [-4.4, -0.2].

So more reasoning is not universally useless. It is model-dependent. But the outcome decomposition reveals a deeper pattern.

Figure 9. Positive exact success and policy-sensitive/completeness correctness are better. Positive extra side effects are worse. For weighted failure loss, negative is better.

Relative to Sol medium, Sol xhigh is associated with +1.3 points on the policy-sensitive/completeness score, -1.3 points on no-extra-state performance, and +1.0 point in extra side effects. The net exact-pass change is almost zero.

Relative to Terra medium, Terra xhigh is associated with +2.9 points on the policy-sensitive/completeness score and +2.5 points in exact success, but also -1.4 points on no-extra-state performance and +1.6 points in extra side effects.

The xhigh configurations are associated with better policy-sensitive/completeness outcomes and more opportunities to touch the system. Reasoning-token count is post-treatment, so this is an effort-configuration comparison, not a per-token causal effect.

The event ledger tells the same story.

Figure 10. Sol xhigh is accompanied by more extra records, attempted value changes, duplicate calls, and read-write continuation without a supported success gain. Terra xhigh improves success, while side effects still rise.

Sol xhigh adds extra records (+1.3 points) and reversals (+1.0) with no supported increase in target read-back and no aggregate success gain. Terra xhigh improves success and reduces tool errors, yet also increases extra records (+1.4) and reversals (+1.6).

The striking conclusion is not “thinking hurts.” Terra disproves it.

It is this:

Xhigh is associated with stronger policy-sensitive/completeness outcomes without guaranteeing control of the external state.

For Sol, the higher policy-sensitive/completeness score is almost exactly cancelled in exact success by more side effects. The xhigh configuration changes which tasks pass without raising the total.

So what: reasoning effort is a deployment configuration, not a universal safety margin. Sol xhigh spends more and changes the failure mix without raising total success; Terra xhigh gains, but still creates additional side effects.

Why public benchmarks and enterprise workflows disagree

There is no contradiction between the public results and ours once the evaluation contract is explicit.

Our deduplicated public ledger contains eight standalone independent lanes. Seven of the eight are run by Artificial Analysis — only CursorBench is not — so “independent” means independent of OpenAI, not eight independent evaluator organizations. We exclude composite indices, provider repetitions, and incompatible benchmark versions. We never pool Elo, pass@1, partial-credit objective share, and other native metrics.

At peak reported configurations:

• Sol is above GPT-5.5 on 7 of 8 lanes.

• Terra is above GPT-5.5 on 6 of 7 populated lanes.

• Both are below GPT-5.5 on tau2 Telecom: GPT-5.5 93.9%, Terra 86.3%, Sol 85.1%.

The tau2 result matters because it is another strict stateful exception. It does not prove a universal class of regression. tau3 Banking slightly favors GPT-5.6. The point is narrower: broad agentic capability and exact workflow control can move in different directions.

Nor is the divergence an artifact of the instrument drifting away from the market. Two checks pin this down, one internal and one external. Internally, every other within-family upgrade the panel has graded moves up — eleven of eleven, from seven vendors, under the exact paired estimand on the same 711 tasks, with GPT-5.4 to GPT-5.5 itself at +3.0 and +3.3 points across both efforts.

Externally, the panel tracks the public consensus model by model: across seventeen fleet models with a published Artificial Analysis Agentic Index score, panel and index agree at Spearman 0.90.

Figure 11. Each point is one model: public Agentic Index (peak presets) against private panel success. The fit and the shaded ±1 and ±2 residual bands use the seventeen non-GPT-5.6 models. The GPT-5.6 configurations are red. AA presets do not always match our arena presets; the effort match is exact for GPT-5.5 xhigh and Sol xhigh.

This figure also answers the most natural question about this article: if GPT-5.6 regressed, why does no public leaderboard show it? Because the effect cannot be seen at that resolution. Cross-vendor level differences scatter about ±9 points around the fit — individual models miss it by up to 21 — an order of magnitude more than the 4-to-9-point within-family regression. At fleet level, the GPT-5.6 points sit inside the band, a fraction below the line, indistinguishable from ordinary vendor-to-vendor variation. Only a paired design — the same 711 tasks, before and after, differenced within task — resolves an effect of this size. Public leaderboards and this article do not disagree about GPT-5.6; they measure at different resolutions.

The internal table also supplies the right scale for the headline numbers: frontier upgrade steps on this panel run about +3 to +7 points, which makes Sol’s regression roughly one upgrade step handed back, and Terra’s more than two.

Coding and terminal benchmarks often reward reaching a valid solution through a long action horizon. Enterprise workflows add a different constraint: the agent must do the authorized thing, touch only the intended objects, preserve invariants, and stop.

GPT-5.6 appears stronger at the first problem. On our panel, it is weaker at the second.

So what: public agent benchmarks and stateful workflow evaluations answer different purchasing questions. A model can lead on broad agency while being a worse drop-in replacement for production systems that require exact, minimal state changes.

What this changes in deployment

1. Do not upgrade agents with a leaderboard

A generic agent benchmark is evidence about broad capability, not a migration test. Re-run the actual workflows, keep repeated trials, and examine the full 0/5 to 5/5 transition matrix. A small average can hide tasks that moved from reliable to broken.

2. Grade the state, not the explanation

Many of the failures in this study produce plausible prose and accepted tool calls. The bug is one extra record, a reversed field, an omitted lifecycle transition, or a write after the supposedly final check. Natural-language judging is poorly positioned to catch that.

3. Make true verification terminal

“Read the record again” is not enough. A read-back event does not prove that the model evaluated the returned state correctly. The agent needs a machine-checkable invariant and a rule that no further mutation occurs after that invariant passes.

4. Treat reasoning effort as a model-specific configuration

Terra benefits from xhigh. Sol does not on exact workflow success. The same token increase is associated with different error-mix changes. Reasoning settings require the same validation as model versions.

5. Instrument side effects directly

Track duplicates, attempted value changes, target read-back, read-write continuation, and unexpected records. These are leading indicators that an agent is solving the task while losing control of the system.

6. Keep the old model in the migration plan

GPT-5.6 fixes real GPT-5.5 failures. GPT-5.5 retains real strengths. Route, canary, and retain rollback coverage until the regression set is understood. “Newer” is not a sufficient workflow policy.

The skeptic’s checklist

A result that contradicts consensus has to earn its keep. This is the complete attack surface as we see it — every objection we could construct against our own result, what the evidence says, and where to check it

The last row is the honest residual. Everything above it has been tested and quantified.

What we know, and what we do not

We know that all four matched GPT-5.6 configurations score lower on this frozen 711-task enterprise panel. The paired effect is supported under task and whole-task-family resampling, remains negative under leave-one-domain-out, and survives alternative reliability and state-quality outcomes.

The visible harness, tasks, policies, tools, grading configuration, and sampling settings define the fixed comparison contract. None of the five identified grading-artifact candidates explains the aggregate regression.

We know that GPT-5.6 performs more target-specific read-back in aggregate (broad-based for Terra, logistics-concentrated for Sol), avoids more airline schema errors, and produces more duplicate calls, attempted value reversals, read-write continuation, and extra records in the structured diagnostics. Those five state-control event families survive correction across 84 declared event tests. The source-turn review and final-state decomposition are consistent with that pattern.

We do not know how much of the observed change comes from model weights, hidden provider snapshots, or another unobserved cross-window runtime factor. We do not claim causal mediation from the trajectory features. The selected transcript audit explains high-contrast cases rather than estimating their population frequency. And the user simulator is itself a model that responds to the agent, so dialogue shape is a joint outcome rather than a fixed stimulus beyond the first turn — the flip review quantifies where that matters and shows the strong flips are reproducible across repeated runs regardless.

Those limits do not erase the operational result. They define it.

GPT-5.6 is not a failed release. It is a broader agent with a different control problem.

It got better at acting.

The next frontier is knowing when to stop.

Methods in brief

• Private panel: 711 identical tasks across six domains, five runs per model-task cell.

• Focal configurations: GPT-5.5 medium/xhigh; Sol medium/xhigh; Terra medium/xhigh.

• Primary estimand: equal-domain average of paired task-cell success differences.

• Uncertainty: 5,000-replicate domain-stratified task bootstrap; whole-family cluster bootstrap over 86 outcome-blind task families; paired task-label randomization; Holm correction over eight frozen contrasts; exact sign randomization over six domain effects reported separately.

• Alternative outcomes: >=4/5, exactly 5/5, no extra state, policy-sensitive/completeness correctness, fixed weighted failure loss.

• Difficulty analysis: within-domain quintiles estimated from four nonfocal context models.

• Event ledger: grade-blind streaming parser over 21,330 trajectories with frozen semantic tool registry and source pointers; BH-FDR over 84 diagnostic tests; 200-row structural/privacy validation plus deterministic semantic-rule checks.

• Failure anatomy: mutually exhaustive fractional attribution of deterministic state diffs; descriptive, not mediation.

• Transcript audit: 240 purposively selected paired cases; 60 independently double-labeled; 95.0% exact agreement, kappa 0.903.

• Flip review: all 153 strong flip cells (123 regressions, 30 fixes) reviewed end to end by model reviewers in two independent passes (81% grouped-label agreement); every 4th regression review adversarially re-derived (61 checks, 0 refuted); artifact candidates disclosed with exclusion sensitivities.

• Public evidence: deduplicated source ledger preserving native benchmark units and effort-setting differences.

• Upgrade concordance: eleven non-focal within-family upgrades recomputed with the primary estimand on the same 711 tasks; all positive with intervals above zero.

• Cross-instrument concordance: panel versus AA Agentic Index across seventeen fleet models, Spearman 0.90; fleet residual scale ±9 pp establishes why the regression is resolvable only in a paired design.

The full technical appendix, machine-readable results, derived content-free trial table, scripts, figures, and verification commands can be downloaded here.

Sources

• OpenAI: GPT-5.6 release

• OpenAI: GPT-5.5 release

• Artificial Analysis: GPT-5.6 independent evaluation

• Artificial Analysis: Agentic Index

• Artificial Analysis: tau2 Telecom

• Artificial Analysis: ITBench-AA

• Artificial Analysis: GDPval-AA

• Artificial Analysis: AA-Briefcase

• Artificial Analysis: AutomationBench-AA

• Artificial Analysis: tau3 Banking

• Artificial Analysis: Terminal-Bench 2.1

• CursorBench 3.2

• Toloka Arena: independent evaluation of agentic intelligence

• Agents do not have a capability problem. They have a comprehension problem.

• Fable 5 reset the leaderboard. The blind spots did not move.