Control tasks
Control tasks contain samples of correct task responses. They allow you to monitor the quality of task completion without having to check all the task responses. Control tasks simplify the verification process, help identify Tolokers' qualifications, and save your time.
To save time, you can opt for having your control tasks edited by Toloka experts. In this case, markup is performed within the pool.
When to use
Use control tasks to assign a skill to Tolokers based on their task responses and ban Tolokers who submit incorrect task responses.
Don't use it if:
- You have a lot of task response options.
- Tolokers need to attach a file to their assignment.
- Tolokers need to transcribe text.
- Tolokers need to select objects in a photo.
- Tasks don't have a correct or incorrect task response. For example: “Which image do you like best?” or “Choose the page design option that you like best”.
How to configure
The If and then fields in this rule are required. If you don't fill in at least one field, you won't be able to save the rule.
| Field | Overview |
|---|---|
| Recent control and training task responses to use | The number of the Toloker's last control task responses. If this field is not filled in, the calculation includes only control task responses in the pool to which the rule applies. If the field is filled in, the corresponding number of control task responses is used. The rule takes into account task responses from both the current pool and other pools where this field is filled in. Learn more about how this field works. |
| If | A condition for performing the action in the then field:
To add multiple conditions, click |
| then | Action to perform for the If condition:
|
.Examples
Purpose: filter out Tolokers who often make mistakes.
Solutions:
-
Ban Tolokers based on control tasks and the percentage of correct task responses
-
Set a skill and deny access to Tolokers with a low skill level
-
Ban Tolokers if their percentage of correct control task responses is less than 40%
The assignments submitted by banned Tolokers will be taken into account if you don't reject them using manual review. They can be reassigned by setting up the Recompletion of assignments from banned users rule.
Ban Tolokers based on control tasks and the percentage of correct task responses
Both rules work independently:
-
If the Toloker gives at least 3 control task responses, the percentage of correct answers is assigned as the skill value. This is useful if you want to prevent Tolokers with a low skill level from completing your tasks.
-
If the Toloker gives at least 3 control task responses and the percentage of correct answers is less than 60%, they lose access to the project.
The calculation uses up to 10 of the Toloker's recent responses to the project's control tasks.
Set a skill and deny access to Tolokers with a low skill level
Skills help identify how well Tolokers do your tasks. You can ban Tolokers with low skill levels from completing tasks in your pool or project.
If the Toloker completes 3 control or training tasks, they get a skill. Use the skill value to set access to other pools with filters.
Ban Tolokers if their percentage of correct responses to control tasks is less than 40%
If the percentage of correct control task responses is less than 40%, the Toloker loses access to the project for 30 days.
This rule doesn't take into account responses in the training tasks for banning.
See also
For developers
Troubleshooting
Last updated: July 26, 2023