Why traditional benchmarks fail for agents and how trajectory-based evaluation with human annotation changes the game

Why AI agents need a new kind of evaluation

AI agents built on Model Context Protocol (MCP) don’t just generate text. They take actions: calling tools, querying databases, modifying records, sending messages, and orchestrating multi-step workflows across connected systems. When an agent writes a wrong sentence, it’s an inconvenience. When an agent executes the wrong tool call in a production system, it’s a material consequence.

Traditional LLM benchmarks were designed for a simpler problem: does the model produce the correct output? Benchmarks like MMLU, HumanEval, and GSM8K test knowledge, coding ability, and mathematical reasoning through static question-answer pairs. They tell you whether a model is smart. They do not tell you whether an agent built on that model will behave correctly inside a real system.

The gap between model capability and agent reliability is where most production failures happen. A model might score 95% on a coding benchmark but still select the wrong API endpoint 8% of the time when orchestrating a multi-step customer service workflow. That 8% error rate, invisible in traditional evals, translates to thousands of incorrect actions per day at enterprise scale.

MCP evaluations close this gap. They test agents inside realistic tool-connected environments and analyze the complete trajectory of actions the agent takes, not just the final answer it produces.

What MCP evaluations are

An MCP evaluation tests how an AI agent behaves when it operates inside an environment of connected MCP servers, the same kind of environment it would encounter in production. The agent is given a goal and must achieve it by selecting tools, calling them with correct arguments, interpreting the results, and chaining multiple steps together.

The evaluation captures the complete trajectory: every tool call, every argument passed, every result received, and every decision made along the way. This trajectory is then analyzed using both automated scoring and human expert annotation to produce a diagnostic profile of the agent’s capabilities and failure patterns.

This approach matters because the same final output can be reached through correct or incorrect trajectories. An agent might return the right answer to a customer query, but only because two errors happened to cancel each other out, for example, retrieving the wrong customer record but then misreading it in a way that coincidentally produces the correct refund amount. Traditional output-only evaluation would score this as a success. Trajectory-based evaluation catches both errors.

How MCP evaluations differ from standard agent benchmarks

The research community has produced several MCP-specific benchmarks in 2025-2026, each approaching the problem from a different angle.

MCP-AgentBench tests agents across 33 MCP servers offering 188 tools, with 600 queries spanning six categories of interaction complexity, from single-server operations to multi-server sequential workflows. It introduced MCP-Eval, an outcome-oriented scoring methodology focused on task completion.

MCP-Universe by Salesforce evaluates agents across six domains with real-world servers, focusing on cross-domain performance variations and revealing that even state-of-the-art models show markedly different success rates across different application domains.

MCP-Bench by Accenture provides a multi-faceted framework covering tool-level schema understanding, trajectory-level planning, and task completion. Testing across 20 advanced LLMs revealed persistent challenges in complex real-world tasks.

MCPEval introduces fully automated evaluation with programmatic task generation, moving beyond the manual task creation that limits scalability in other frameworks.

These benchmarks share a common limitation that an ICLR 2026 blog post articulated clearly: MCP itself does not offer built-in support for evaluation workflows. It has no standard way to represent tasks, report metrics, track experiments, or ensure reproducibility. Any evaluation framework built on MCP is essentially defining an additional protocol on top of it.

Toloka’s approach to MCP evaluations addresses this by combining proprietary environments with expert human annotation, creating a diagnostic layer that goes beyond what automated scoring alone can provide. The methodology draws on Toloka’s work on tau-bench, which demonstrated that agents behave differently when users interact like real people rather than scripted prompts.

The Toloka MCP evaluation framework

Toloka’s MCP evaluations run agents inside proprietary environments that mirror real operational systems. Each environment includes the types of tools, data structures, and constraints an agent encounters in production: a manufacturing operations system with Zendesk tickets and escalation policies, a marketplace assistant with product catalogs and return rules, or a financial services workflow with compliance checks and approval gates.

These environments are not found in public training data. When an agent improves its score across evaluation sprints, that improvement reflects genuine capability growth, not memorization of benchmark answers.

Hybrid scoring: automated rewards + human annotation

Every evaluation run produces two layers of signal. Automated rewards check objective criteria: did the agent call the right tool? Were the arguments valid? Did it complete the task? Human expert annotators then review the full trajectory to catch subtler issues: did the agent follow the correct reasoning path even when it reached the right answer? Did it respect policy boundaries? Did it handle ambiguity appropriately?

This hybrid approach is critical because automated scoring misses a category of failures that only human judgment can identify. An agent might construct a technically valid API call that violates an unwritten business rule. It might retrieve the correct data but draw an incorrect inference from it. It might follow the right sequence of steps but use the wrong reasoning to justify them.

The 12-type failure taxonomy

When an agent fails, the diagnosis matters as much as the detection. Toloka’s annotators classify every failure using a structured taxonomy covering 12 distinct error types across three categories:

Tool execution faults. The agent selected the wrong tool for the task, passed invalid or malformed arguments, called tools in an incorrect sequence, or failed to use a required tool entirely.

Data grounding issues. The agent misread or misinterpreted data returned by a tool, confused one entity with another (for example, mixing up two customer records), missed required fields in the returned data, or cited data that was never actually retrieved.

Reasoning failures. The agent lost track of constraints across multiple steps, applied a policy incorrectly, made unsupported logical inferences, or revealed gaps in domain knowledge that led to wrong conclusions.

This taxonomy transforms evaluation from a passive score into an actionable diagnostic. Instead of knowing that an agent failed 12% of tasks, a team sees that 40% of failures are tool-sequencing errors, 35% are data misinterpretation, and 25% are policy violations. Each category points to a different training fix.

Statistical rigor: sprints and multirun analysis

A standard Toloka evaluation sprint covers 300 to 500 proprietary data points across multiple environments. Every data point is run multiple times to establish a reliable failure rate rather than relying on a single pass. Results are enriched with metadata: domain, specific policy tested, user intent category, complexity level.

Teams typically start with a single sprint to benchmark their current agent, then move to weekly cadence during active training or fine-tuning. Each sprint produces a detailed report combining automated metrics with the human-annotated failure breakdown, creating a longitudinal view of capability development that catches regressions early.

Connect your AI agent to human experts

Tendem MCP gives your agent access to 10,000+ vetted domain specialists on demand. One install, no code changes, non-blocking async execution.

Learn more about Tendem MCP →

A practical example: where trajectory evaluation catches what output evaluation misses

Consider a customer service agent deployed in a retail environment. A customer asks to return an item purchased three weeks ago. The correct workflow involves several steps: verify the customer’s identity, retrieve the order details, check the return policy (30-day window, receipt required, certain categories excluded), process the return if eligible, and update the ticket.

In a traditional output-focused evaluation, you would check: did the agent correctly approve or deny the return? If the answer matches the expected outcome, the agent passes.

In a trajectory-based MCP evaluation, you examine the entire path:

Agent A retrieves the wrong order (tool execution fault: wrong arguments passed to the order lookup tool), but the wrong order also happened to be within the return window. It approves the return. Output: correct. Trajectory: critically flawed. In production, this agent would process returns against wrong orders.

Agent B retrieves the correct order but skips the category exclusion check (reasoning failure: lost constraint). The item happens to be in a returnable category. Output: correct. Trajectory: unsafe. In production, this agent would approve returns on excluded categories.

Agent C follows the full workflow correctly but approves the return without updating the ticket (tool execution fault: missing required action). Output: partially correct. Trajectory: incomplete. In production, this agent would leave tickets unresolved.

All three agents produce a plausible output. Only trajectory evaluation reveals the specific failures that would cause problems in production. And crucially, each failure points to a different fix: Agent A needs better tool argument construction, Agent B needs policy constraint retention, Agent C needs workflow completion verification.

Toloka Arena: the public leaderboard

For teams that want to compare model capabilities before committing to a full evaluation engagement, Toloka Arena provides an independent leaderboard for agentic intelligence. It evaluates leading LLMs on private, non-contaminated tasks across multiple domains using simulated real-world customer service scenarios inspired by the tau-bench methodology, with live databases, API tools, and strict business rules.

The leaderboard uses a composite pass^5 score (the probability of passing a task in five consecutive attempts) which penalizes inconsistent agents more heavily than a simple pass rate. It also plots score against average inference cost per task, helping teams identify the best performance-to-cost tradeoff for their use case.

The benchmarks powering the leaderboard are available for licensing. Teams can use Toloka’s RL Gyms and evaluation data to train and test their own models, and the tau-bench dataset extension is available for policy-aware agent evaluation in manufacturing and other constrained environments.

Who needs MCP evaluations

Model providers

For teams building foundation models and agent frameworks, MCP evaluations provide the high-resolution diagnostic signal needed to improve tool-calling capabilities systematically. Instead of a generic "fail," providers see whether the model needs stronger tool argument construction, better long-context constraint retention, or more accurate policy adherence. Because environments are proprietary and updated continuously, providers can be certain that performance gains reflect genuine capability rather than benchmark memorization. These evaluations serve as practical training signal, not just a report card. For teams also benchmarking coding agents, our analysis of SWE-bench limitations covers similar themes of trajectory versus output evaluation.

Enterprise teams

For organizations deploying agents into production systems, MCP evaluations de-risk the transition from prototype to production. They surface failures, like an agent escalating a ticket without checking policy approval, before those failures impact real users or internal systems. By understanding failure rates across multiple runs, teams determine whether a model is reliable enough for high-stakes tasks or needs additional guardrails. Domain-specific environments can be configured to mirror the exact tools, policies, and workflows an agent will face in production.

The production readiness gap

Gartner projects that 40% of enterprise applications will include task-specific AI agents by end of 2026, but also warns that over 40% of agentic AI projects could be canceled by 2027 due to escalating costs, unclear value, and inadequate governance. The gap between adoption intent and production readiness is the defining challenge, and evaluation is the bridge. Teams that invest in trajectory-based evaluation during development are significantly more likely to reach production successfully.

Where human expertise fits in the evaluation stack

MCP evaluations reveal how agents fail. Human expertise prevents failures from reaching production in the first place.

Tendem by Toloka provides a complementary layer: an MCP server that connects agents to 10,000+ vetted domain experts who handle tasks requiring judgment the agent cannot provide. When evaluation reveals that an agent consistently fails on policy-sensitive decisions, the solution is not always more training. Sometimes the right architecture is to have the agent escalate those specific decisions to a human expert via Tendem, while handling everything else autonomously.

In Tendem benchmarks across 94 real-world tasks, this hybrid approach achieved 1.8x higher quality than AI-only execution. The combination of MCP evaluations (revealing where agents fail) and Tendem (providing human expertise where they fail) creates a closed loop: evaluate, identify weak points, add human oversight where needed, re-evaluate to confirm improvement.

See how your agent really performs

Toloka’s MCP evaluations reveal where AI agents fail in real workflows, with trajectory analysis and human-annotated failure reports.

Talk to us →

Frequently asked questions

What is an MCP evaluation?

An MCP evaluation tests how an AI agent behaves inside realistic, tool-connected environments built on the Model Context Protocol. Instead of checking whether the agent produces a correct final answer, it evaluates the full trajectory: which tools the agent called, in what order, with what arguments, and whether it interpreted the results correctly.

How is an MCP evaluation different from benchmarks like SWE-bench or GAIA?

SWE-bench and GAIA test models on specific output correctness (code patches, factual answers). MCP evaluations test agents on full workflow behavior inside multi-tool environments. They capture not just whether the task was completed, but whether the agent followed the correct sequence of tool calls, respected policies, and maintained consistent reasoning across steps.

What does Toloka’s 12-type failure taxonomy cover?

The taxonomy classifies failures into three categories with 12 specific types. Tool execution faults cover wrong tool selection, invalid arguments, improper sequencing, and missing tool calls. Data grounding issues cover misreading data, mixing entities, missing fields, and citing unretrieved data. Reasoning failures cover lost constraints, incorrect policy application, unsupported inferences, and domain knowledge gaps.

How many data points does a typical MCP evaluation sprint cover?

A standard sprint covers 300 to 500 proprietary data points across multiple environments. Each data point is run multiple times (multirun analysis) to establish reliable failure rates. Smaller focused sprints using off-the-shelf environments like the TAU-inspired manufacturing dataset can cover 50 data points for a deep dive into specific workflow types.

Can I evaluate my own AI agent with Toloka’s MCP evaluations?

Yes. Toloka offers MCP evaluations as a managed service for model providers and enterprises. You can start with off-the-shelf environments (manufacturing, marketplace, customer support) or build custom environments matching your production systems. Toloka Arena provides a public leaderboard for comparing model capabilities before committing to a full engagement.

Related reading

The importance of MCP evaluations in agentic AI

Tau-bench: the next generation of AI agent evaluation

Tau-bench extension: benchmarking policy-aware agents

AI agent evaluation methodologies, challenges, and emerging standards

Does your agent work? AI agent benchmarks explained