Control tasks

Control tasks contain samples of correct task responses. They allow you to monitor the quality of task completion without having to check all the task responses. Control tasks simplify the verification process, help identify Tolokers' qualifications, and save your time.

Note

To save time, you can opt for having your control tasks edited by Toloka experts. In this case, markup is performed within the pool.

When to use

Use control tasks to assign a skill to Tolokers based on their task responses and ban Tolokers who submit incorrect task responses.

Don't use it if:

  • You have a lot of task response options.
  • Tolokers need to attach a file to their assignment.
  • Tolokers need to transcribe text.
  • Tolokers need to select objects in a photo.
  • Tasks don't have a correct or incorrect task response. For example: “Which image do you like best?” or “Choose the page design option that you like best”.

How to configure

Alert

The If and then fields in this rule are required. If you don't fill in at least one field, you won't be able to save the rule.

FieldOverview
Recent control and training task responses to useThe number of the Toloker's last control task responses.

If this field is not filled in, the calculation includes only control task responses in the pool to which the rule applies.

If the field is filled in, the corresponding number of control task responses is used. The rule takes into account task responses from both the current pool and other pools where this field is filled in.

Learn more about how this field works.
If

A condition for performing the action in the then field:

  • number of task responses — The number of completed control and training tasks.

  • correct task responses (%) — The percentage of correct task responses in training and control tasks (from 0 to 100).

  • incorrect task responses (%) — The percentage of incorrect task responses in training and control tasks (from 0 to 100).

  • number of control task responses — The number of completed control tasks.

  • correct control task responses (%) — The percentage of correct task responses in control tasks (from 0 to 100).

  • incorrect control task responses (%) — The percentage of incorrect task responses in control tasks (from 0 to 100).

To add multiple conditions, click .

then

Action to perform for the If condition:

  • assign skill from the field — Save the percentage of the Toloker's correct task responses in control tasks as a skill value.

  • assign skill value — Assign a fixed value to the Toloker's skill.

  • accept all assignments from this Toloker in the pool — Requires the manual review option to be set.

    Useful if the Toloker completes most tasks well. Example: The Toloker completed more than 80% of the tasks correctly and you're satisfied with this result. The rule will work automatically and accept all task suite responses in the pool.

  • suspend — Suspend the Toloker's access to the pool for the specified number of days. Only the requester can view the reason.

  • ban — Block access to the project or all of the requester's projects for the specified number of days. Only the requester can view the reason.

    If access to tasks is blocked temporarily (for example, for 7 days), the history of the Toloker's responses is not saved after the ban is lifted. The skill level is calculated based on the new responses.

Examples

Purpose: filter out Tolokers who often make mistakes.

Solutions:

Warning

The assignments submitted by banned Tolokers will be taken into account if you don't reject them using manual review. They can be reassigned by setting up the Recompletion of assignments from banned users rule.

Ban Tolokers based on control tasks and the percentage of correct task responses

Correct settings
Incorrect settings
Alternative settings

Both rules work independently:

  1. If the Toloker gives at least 3 control task responses, the percentage of correct answers is assigned as the skill value. This is useful if you want to prevent Tolokers with a low skill level from completing your tasks.

  2. If the Toloker gives at least 3 control task responses and the percentage of correct answers is less than 60%, they lose access to the project.

The calculation uses up to 10 of the Toloker's recent responses to the project's control tasks.

Skills help identify how well Tolokers do your tasks. You can ban Tolokers with low skill levels from completing tasks in your pool or project.

Correct settings
Incorrect settings

If the Toloker completes 3 control or training tasks, they get a skill. Use the skill value to set access to other pools with filters.

Example of filter settings

Ban Tolokers if their percentage of correct responses to control tasks is less than 40%

Correct settings
Incorrect settings

If the percentage of correct control task responses is less than 40%, the Toloker loses access to the project for 30 days.

This rule doesn't take into account responses in the training tasks for banning.

See also

For developers

Troubleshooting

How many control tasks do I need to add?
How are the correct control task responses counted?
Should I create a skill for every pool?
Can I use a skill beyond a particular pool or project and apply it to other projects as well?
How do I move control tasks from the sandbox to the main pool?
If a cheating Toloker gives a lot of incorrect responses, and the system eventually bans them for errors in control tasks, do I have to pay for the bad responses anyway?
When I export a project from the sandbox, the task files are not exported. Is this how it's supposed to work? I suddenly lost the markup of the control tasks that I created in the sandbox.
If I upload tasks using smart mixing, does it mean that the same file should contain both the control tasks and general tasks, or can I upload them separately?
Contact support

Last updated: July 26, 2023

Introduction
Getting started
Important tips
Useful recommendations
Working with Toloka
Projects
Pools
Tasks
Results
Project analysis
Toloka settings
Task interface
Template Builder
HTML/CSS/JS editor
Help and support
FAQTroubleshootingSupportGlossary