How Shopify and Toloka built a ground-truth flywheel to keep skill agents accurate at scale

Shopify's Sidekick, an AI-enabled commerce agent, helps merchants run their stores through plain language conversations, whether they’re asking for a sales workflow or want to build a new storefront app. Behind its simplicity is a planner model, routing requests to a growing bench of specialized skill agents, such as business analytics, workflow generation, web app generation, image generation, and more, with new skills added on an ongoing basis.

If you’re part of an engineering team building agentic systems, it’s easy to underestimate that every new skill needs its own quality signal. A skill agent and its judge can look strong in eval yet still lose accuracy once it meets the long tail of real merchant phrasing, ambiguous time windows, and edge cases no test set anticipated. The quicker you ship new skills, the faster that signal needs to compound — or your fastest-growing surface area becomes your least-monitored one.

This is the case study of how Shopify and Toloka built a reliable quality signal, and what it took to take a skill-judge from an initial evaluation baseline to a level Shopify could trust as production ground truth.

Challenge

Sidekick's skill agents follow a three-level structure: a planner (or orchestrator) that acts as the conversation layer, the specific skill itself, and a judge model that not only evaluates output quality but also attempts to heal trajectories it deems incorrect. 

Take customer segmentation, one of the first skills Toloka worked on. A merchant asks something like "create a segment for customers who spent $500 or more last month," and the agent translates that into a precise, executable query.

The gap between "this sounds right" and "is right" here is small and easy to miss without a thorough review. A query can run and return data but still miss the merchant's intent — resolving a loose product title match instead of using an available exact ID, hard-coding a relative date that breaks next year, or dropping part of a compound request. None of these show up as errors. But they do show up as merchants losing trust in an answer that looks plausible.

Shopify needed a way to catch failures like this at the pace it was shipping new skills, generate the training data to fix them, and build enough confidence in the judge evaluating each skill that evaluation itself could scale without a human reviewing every conversation forever.

Solution

Toloka provides the human labelling behind Shopify's model-distillation flywheel. De-identified production conversations flow through the skill agent and are scored by an LLM judge. Anything below threshold — after an automated healing attempt — routes to Toloka for human correction before it's used for fine-tuning or sent back into the loop to re-score the judge itself. As Shopify's engineering team put it in their own recent post on the flywheel: "if it still fails, we flag it for human annotation... that’s where Toloka comes in."

That work runs across two streams:

• Zero Scores (ZS): Toloka experts imitate the work the judge has done by giving the verdict and suggesting remediation strategies, turning these into SFT (Supervised Fine-Tuning) golden sets for the skill model.

• Thumbs Down (TD): Toloka analyzes conversations merchants themselves flagged to capture real-time dissatisfaction signals. Acting as an analytical support layer, Toloka detects spikes in merchant complaints following new skill updates or releases, and escalates targeted issues to specific Shopify teams so they can swiftly investigate and resolve root causes.

Both streams rely on the same underlying discipline: a detailed, versioned annotation guideline, a skill-specific failure-mode taxonomy, and BI/SQL-literate reviewers matched to the task, as well as coding experts for workflow-related tasks.

The more difficult problem was trust delegation for the judge itself. Shopify's own team is the ground truth. Each week, Toloka's annotations go through Shopify review on a sample of key labels. The sample rate starts high and tapers as agreement holds. Every disagreement is traced back to its root cause: is it a genuine miss, a defensible read of an ambiguous guideline, or a completely new edge case?

Annotation quality is tracked explicitly as one minus the disagreement rate, and guidelines are updated the moment a real gap surfaces. Only once that annotation layer clears a high bar does it get used to build the ground truth set (GTX) the judge itself is measured against. In practice, the GTX functions as a spec: it captures both good and bad scenarios and the corner cases in between, each scored against product-specific criteria the judge is expected to apply. A judge trained on shaky GTX produces a more convincing wrong answer, faster.

The intent from the start was for none of this to be bespoke per skill — the same trust-delegation loop, failure-mode framework, and expert-matching logic would need to hold up as Sidekick's skill roster continues to grow. 

Impact

At a Glance

• Boosted Accuracy: The segmentation skill-judge's average class accuracy moved from 75% to 89% in just a month — through prompt engineering alone, without a single retraining cycle.

• Direct Model Improvement: With a trusted judge in place, Toloka's corrected answers went straight into fine-tuning the skill models themselves.

• Actionable Analytics: Root-cause analysis named the two specific failure modes driving low quality (incorrect tool selection and unnecessary opt-outs), allowing Shopify to focus engineering efforts where they mattered most.

• Cost Efficiency: Toloka cut per-task annotation costs by roughly half through experience-curve gains and pipeline automation.

How We Got There

The segmentation skill-judge's average class accuracy moved from 75% to 89% within a month. Because Toloka's experts evaluated the tasks entirely blind — imitating the judge without seeing its actual verdicts — this accuracy gain represents a genuine alignment with ground truth. It crosses the point where a judge's calls can be trusted at scale rather than double-checked by hand.

With a judge Shopify could trust, Toloka's corrected answers became training signal. Following Shopify's own model-distillation approach, those corrected trajectories — reasoning and all, not just final answers — went into supervised fine-tuning to teach the segmentation and business analytics skill models the corrected behavior directly. These then went into reinforcement learning, using the calibrated judge itself as the reward signal. Output quality improved across active skills, with stronger results showing up in the work itself.

Root-cause analysis from the annotation layer did what dashboards alone couldn't, naming the two failure modes driving low segmentation and analytics quality: incorrect tool selection and unnecessary opt-out calls. This allowed Shopify to direct its engineering effort toward the specific failures bringing down quality.

• Incorrect tool selection showed up when the agent routed a request to the wrong skill instead of engaging with what the merchant actually asked for, returning, for example, adjacent reports rather than the specific guidance requested.

• Unnecessary opt-outs looked safer but created the same downstream cost (and merchant frustration). The agent had a resolvable path for the request — sometimes even one it had just used successfully — and declined it anyway. In one example, after already returning a transaction breakdown by both location and payment type in one query, the agent refused the action and asked the merchant to pick just one of those dimensions to continue instead of simply extending the same combined view.

Just as engineering efforts became more targeted, that same discipline applied to spend: Toloka cut per-task annotation costs by roughly half through experience-curve gains and pipeline automation, ensuring the cost of maintaining quality doesn't scale linearly with the number of skills Sidekick ships.

This flywheel isn't unique to segmentation or business analytics. Shopify has published details of how the same continual-learning loop works across its Sidekick skills, including how it has helped their GraphQL agent surpass frontier-model quality, while separately cutting serving costs by 96%. It's the same underlying flywheel that Toloka and Shopify are extending across the rest of Sidekick's growing skill set.