To improve quality of answers, you must select the audience you need and train it.

Selecting the audience


  • Demographics (age, gender, education, languages, region, citizenship).

  • Device specs (device type, OS and browser version).

  • Skills (quality on specific tasks).

  • Top % of the best on the platform.

Filter example

Training and onboarding

  • Correct answers + hints.

  • High scores continue on to the exam.


  • Scored by % correct answers.

  • Best scores grant access to paid tasks.

Multi-level quality checks

Quality control

Anti-fraud system

  • Platform-wide ban for fraudulent Tolokers.

  • Behavior analysis system.

  • Multilayer technologies to detect and prevent all types of fraud.

Protect your project

Toloka has dedicated anti-fraud system for banning cheaters, but the quality control is shared responsibility of the requester and the platform. The requester is responsible for the quality control of his projects and protection of his data. Projects require individual approach in setting quality controls to ensure best quality of labelled data.

To protect your project from cheaters, you can use the quality control rules:

Examples of the quality control rules

Obtaining quality results after data labeling

Aggregating results

Evaluating metrics

  • Accuracy/completeness/F1/MCC, etc.

  • Consistency.

  • Confidence.

Reassigning tasks

  • If the submitted task is rejected.

  • If consistency is low.

  • If answers from banned Tolokers are thrown out.

Main project scenarios

Validate results

  • Control tasks — set the Toloker’s skill level based on answers in control tasks and exclude Tolokers who give the wrong answers.

  • Majority vote — have multiple Tolokers do the same task and look for consistency in answers.

  • Manually check results — evaluate Tolokers by the number of accepted and rejected tasks.

Diversify the audience

  • Earnings — limit the earnings per person in your pool in 24 hours.

  • Completed tasks — limit the number of tasks per person in your pool in 24 hours.

Prevent random clicking and bots

Redo certain tasks

  • Re-assign tasks completed by someone who was banned — if a Toloker gets banned, all their completed tasks can be automatically assigned to other people.

  • Rejected and accepted task processing — set the rules for assigning rejected tasks to other people.

Technologies for quality management

Transform the crowd into computing power with advanced technologies for quality management.

Multiple quality control methods

Toloka offers different approaches to achieve the best quality for each project.

  • Post-verification.

  • Task-based crowd training and testing.

  • Golden sets (honeypots) to monitor quality.

  • Advanced aggregation tools.

  • Platform-wide anti-fraud system.

Adaptive selection of Tolokers

Multi-stage selection of a distributed crowd.

  • Audience filters by language, age, gender, interests, location, real-time ranking, and more.

  • Training, exams, and retraining to find Tolokers for your exact task.

Smart matching mechanisms

Patent-pending matching system that honors the preferences of requesters and Tolokers for mutual benefit.

  • Invite Tolokers to a project who are most qualified to handle it.

  • Offer Tolokers personalized recommendations of interesting projects they will enjoy.

Autolabeling & verification (on demand)

Autolabeling and pretrained models with quality control built in.

  • Automated prelabeling. Results are verified by human Tolokers for high accuracy.

  • Human in the loop workflows.

What's next

See also


Setting up quality control
Control tasks
Majority vote
Speed of task completion
Contact support

Last updated: February 15, 2023

Getting started
Important tips
Useful recommendations
Working with Toloka
Project analysis
Toloka settings
Task interface
Template Builder
HTML/CSS/JS editor
Help and support