Supporting Tavily’s new benchmark with Toloka’s self-service platform

How much confidence should you have in the information an AI agent finds for you? Tavily’s new Company Search Benchmark looks at that very question for company research, where a wrong financial figure could influence important business decisions. To make sure its reference answers used accurate figures, Tavily ran expert reviews on Toloka’s self-service platform, bringing financial experts into the evaluation without searching for or inviting a single reviewer.

What’s behind the benchmark?

Company research can take some digging, especially when financial information needs cross-referencing against its original source. While an agent might find a figure that looks right, knowing whether it answers the question requires a closer look at the context.

Tavily, the web access layer for AI agents, compares nine search providers in its new benchmark and assesses how much a single search response helps an agent answer a company research question. Before judging the answers, Tavily builds a scoring card for each query and sets out what a correct response should cover based on the available evidence. Those scoring cards are only as reliable as the figures in them, so Tavily turned to Toloka to verify those figures.

Read Tavily’s blog for the full methodology.

Building the evaluation dataset

Tavily built the dataset from recent production query patterns to ensure the benchmark reflects how agents actually research companies. To protect privacy, Tavily removed customer identifiers and replaced company names and other identifying details with ones drawn from public company registries. It kept the mix of jurisdictions, company sizes, and public and private companies the same as in the original queries.

Tavily started with each query instead of a ready-made answer. This kept the hardest questions in the benchmark, including ones with no complete answer on the public web. Because of that, a scoring card can’t be a single correct number. Each card has to define what useful partial information looks like and how to handle cases where the evidence can’t support a full answer.

How Tavily set up the expert review on Toloka

The setup took three steps: 

• Project description: The team described the project in plain language and worked with the platform’s AI assistant to set up the review process, including the task interface and quality criteria.

• Expert matching: Tavily chose the financial specializations the task needed. Behind the scenes, Toloka’s Smart Search ranked verified experts by how relevant their backgrounds were to the task, not just by listed skills, and invited the best fits to the project.. 

• Quality control: An automated check assessed each expert’s submission against the project’s rules, including whether the grade matched the rubric and whether corrections covered every requested metric. Reference examples helped calibrate the checks for cases such as outdated figures and private-company estimates.

What the experts checked

Reviewing financial answers involves more than confirming numbers on a page. The experts were tasked with:

• Checking figures against sources: Experts compared each number against the cited evidence. If a link was broken or didn’t support the figure, they found an alternative and recorded the source. Company filings provided primary evidence, while private-company questions sometimes required estimates from data providers.

• Checking what the numbers represented: An annual revenue figure might be copied perfectly from a reliable source, but it wouldn’t answer a question about quarterly revenue. Reviewers also looked at units and whether figures were quoted directly from a source. Calculating an annual figure from a single quarter wasn’t allowed.

• Grading and correcting answers: Experts graded each answer as fully correct, partially correct, or incorrect. Any answer that fell short of fully correct had to be rewritten in full, covering every requested metric and stating the reporting period. If no source contained a figure, the answer needed to say so.

The reviewed answers gave Tavily verified financial figures for its scoring cards, which every answer in the benchmark is judged against. Tavily launched new batches on its own schedule, several times a week, and refined the review process between batches. Later batches kept the same experts, so they could build on what they’d learned in earlier rounds.

Build your own benchmark with Toloka

If your benchmark needs answers you can trust, Toloka can help. Tavily used Toloka’s self-service platform to bring financial experts into an otherwise automated pipeline, sending them only the part that needed human judgment. You can set up the same kind of expert review for your own evaluation: describe the task, and the platform finds the right experts and checks their work.

Get started with the Toloka Platform.