Control tasks contain samples of correct task responses. They allow you to monitor the quality of task completion without having to check all the task responses. Control tasks simplify the verification process, help identify Tolokers' qualifications, and save your time.
To save time, you can opt for having your control tasks edited by Toloka experts. In this case, markup is performed within the pool.
When to use
Use control tasks to assign a skill to Tolokers based on their task responses and ban Tolokers who submit incorrect task responses.
Don't use it if:
You have a lot of task response options.
Tolokers need to attach a file to their assignment.
Tolokers need to transcribe text.
Tolokers need to select objects in a photo.
Tasks don't have a correct or incorrect task response. For example: “Which image do you like best?” or “Choose the page design option that you like best”.
How to configure
The If and then fields in this rule are required. If you don't fill in at least one field, you won't be able to save the rule.
Recent control and training task responses to use
The number of the Toloker's last control task responses.
If this field is not filled in, the calculation includes only control task responses in the pool to which the rule applies.
If the field is filled in, the corresponding number of control task responses is used. The rule takes into account task responses from both the current pool and other pools where this field is filled in.
A condition for performing the action in the then field:
number of task responses — The number of completed control and training tasks.
correct task responses (%) — The percentage of correct task responses in training and control tasks (from 0 to 100).
incorrect task responses (%) — The percentage of incorrect task responses in training and control tasks (from 0 to 100).
number of control task responses — The number of completed control tasks.
correct control task responses (%) — The percentage of correct task responses in control tasks (from 0 to 100).
incorrect control task responses (%) — The percentage of incorrect task responses in control tasks (from 0 to 100).
To add multiple conditions, click .
Action to perform for the If condition:
assign skill from the field — Save the percentage of the Toloker's correct task responses in control tasks as a skill value.
assign skill value — Assign a fixed value to the Toloker's skill.
accept all assignments from this Toloker in the pool — Requires the manual review option to be set.
Useful if the Toloker completes most tasks well. Example: The Toloker completed more than 80% of the tasks correctly and you're satisfied with this result. The rule will work automatically and accept all task suite responses in the pool.
suspend — Suspend the Toloker's access to the pool for the specified number of days. Only the requester can view the reason.
ban — Block access to the project or all of the requester's projects for the specified number of days. Only the requester can view the reason.
If access to tasks is blocked temporarily (for example, for 7 days), the history of the Toloker's responses is not saved after the ban is lifted. The skill level is calculated based on the new responses.
Purpose: filter out Tolokers who often make mistakes.
Ban Tolokers based on control tasks and the percentage of correct task responses
Both rules work independently:
If the Toloker gives at least 3 control task responses, the percentage of correct answers is assigned as the skill value. This is useful if you want to prevent Tolokers with a low skill level from completing your tasks.
If the Toloker gives at least 3 control task responses and the percentage of correct answers is less than 60%, they lose access to the project.
The calculation uses up to 10 of the Toloker's recent responses to the project's control tasks.
Set a skill and deny access to Tolokers with a low skill level
Skills help identify how well Tolokers do your tasks. You can ban Tolokers with low skill levels from completing tasks in your pool or project.
If the Toloker completes 3 control or training tasks, they get a skill. Use the skill value to set access to other pools with filters.
Example of filter settings
Ban Tolokers if their percentage of correct responses to control tasks is less than 40%
If the percentage of correct control task responses is less than 40%, the Toloker loses access to the project for 30 days.
This rule doesn't take into account responses in the training tasks for banning.