Quality control rules
Quality control rules allow you to get more accurate responses and restrict access to tasks to cheating Tolokers. All rules work independently.
List of rules
-
To keep track of how often Tolokers make mistakes:
-
Control tasks: Use them to assign a skill to Tolokers based on their control task responses and ban Tolokers who submit incorrect responses.
-
Majority vote: Quality is based on matching the task response from the majority of Tolokers who complete the same task.
-
Results of checking: Evaluate Tolokers based on the number of accepted and rejected task suite responses.
-
-
To protect your project from robots and cheaters:
-
Fast responses: Control the minimum time that must be spent per task suite.
-
Skipped assignments: Restrict access to your pool tasks for Tolokers who skip multiple assignments in a row.
-
-
To attract a variety of Tolokers:
-
Earnings: Limit the amount each Toloker can earn in the pool per day.
-
Submitted assignments: Limit how many assignments each Toloker can submit in the pool per day.
-
-
To allow recompletion of certain assignments:
-
Recompletion of assignments from banned users: Send completed assignments to other Tolokers to redo them if the Toloker was banned.
-
Processing of rejected and accepted assignments: Send rejected assignments to other Tolokers to redo them.
-
Quality control presets
Toloka has presets of quality control rules. Currently, there are three of them:
-
Elementary — suitable for requesters who need tasks to be completed by a large number of Tolokers in a short time.
-
Basic — suitable for those who need to balance quality and quantity.
-
Advanced — a set with a large number of rules required for complex tasks.
The table shows the rules included in each of the sets.
| Earnings | Skipped assignments | Control tasks | Majority vote | Submitted answers | Fast responses | |
|---|---|---|---|---|---|---|
| Elementary | + | + | − | − | − | - |
| Basic | + | + | + | + | − | - |
| Advanced | + | + | + | + | + | + |
How to set up quality control
You can configure quality control in the pool and in the project.
Go to pool editing (click 
in the upper-right corner of the pool page) and click Add a quality control rule in the Quality control section.
You can copy quality control settings from another pool. To do this, click Copy audience filters and quality control settings in the Audience section.
See also
For developers
Troubleshooting
Last updated:Â February 15, 2023