Quality control rules
Quality control rules allow you to get more accurate responses and restrict access to tasks to cheating performers. All rules work independently.
List of rules
- To keep track of how often performers make mistakes:
- Control tasks: Use them to assign a skill to performers based on their responses to control tasks and ban performers who submit incorrect responses.
- Majority vote: Quality is based on matching the response from the majority of performers who complete the same task.
- Results of checking: Evaluate performers based on the number of accepted and rejected responses.
- To protect your project from robots and cheaters:
- Fast responses: Control the minimum time that must be spent per task suite.
- Captcha: Show a captcha from time to time to make sure tasks aren't completed by robots.
- Skipped assignments: Restrict access to your pool tasks for performers who skip multiple assignments in a row.
- To attract a variety of performers:
- Submitted assignments: Limit how many assignments each performer can submit in the pool per day.
- To allow recompletion of certain assignments:
- Recompletion of assignments from banned users: Send completed assignments to other performers to redo them if the performer was banned.
- Processing of rejected and accepted assignments: Send rejected assignments to other performers to redo them.
Quality control presets
- Elementary — suitable for requesters who need tasks to be completed by a large number of performers in a short time.
- Basic — suitable for those who need to balance quality and quantity.
- Advanced — a set with a large number of rules required for complex tasks.
| Earnings | Skipped assignments | Control tasks | Majority vote | Submitted answers | Fast responses | |
|---|---|---|---|---|---|---|
| Elementary | + | + | − | − | − | − |
| Basic | + | + | + | + | − | − |
| Advanced | + | + | + | + | + | + |
| Earnings | Skipped assignments | Control tasks | Majority vote | Submitted answers | Fast responses | |
|---|---|---|---|---|---|---|
| Elementary | + | + | − | − | − | − |
| Basic | + | + | + | + | − | − |
| Advanced | + | + | + | + | + | + |
How to set up quality control
You can configure quality control in the pool and in the customizable project.
Go to pool editing (the Edit button in the upper-right corner of the page) and click Add Quality Control Rule.
You can copy quality control settings from another pool. To do this, click Copy settings from in the Users filter section.
Open the project page, open the Quality control tab and click Set quality control. Then click + Add Quality Control Rule.
The rules are applied to all project pools, so you can't change settings in just one of the pools.
When you clone a project, its quality control settings aren't transferred.
Troubleshooting
Always use one or more ways to control quality of answers.
Counting fast responses makes sense for most tasks.
If the user has to choose between options (for example, by selecting checkboxes), check the answers using majority vote or control tasks.
If the user has to provide a response as a text or link or upload a photo, the best way to control quality is by reviewing assignments. You can outsource task acceptance to performers. Create a task with a question (for example, “Is this phrase translated correctly?”) and possible responses (for example, “yes”/“no”). Set up overlap and majority vote check.
If a task is more like an opinion poll (for example, choosing nice pictures from a set), majority vote is not a good way to control quality. Make control tasks with artificial examples where the choice is evident.
Calculate the skill for each pool separately. The current skill value is the value of the skill in the pool the user completed last. This option is convenient if:
The pools are intended for different groups of performers (for example, there are filters by city or country).
Pools are started one by one and you don't want to take into account the responses in the previous pools to calculate the skill in the current pool.
This calculation method is used by default when adding a quality control rule to a pool. For the control tasks block, leave the Recent control task responses to use field empty.
Calculate skill based on all tasks in a project This option is good if the pools are small and you don't need to have skill calculated for each pool.
This option is available only for skills on control tasks. To use it, fill in the Recent control task responses to use field in pool quality control rules.
- No
- Don't show captchas.
- Low
- Show a captcha after every 20 assignments.
- Average/High
- Show a captcha after every 10 assignments.
- No
- Don't show captchas.
- Low
- Show a captcha after every 20 assignments.
- Average/High
- Show a captcha after every 10 assignments.
- You won't be able to correctly evaluate the quality of the performer's responses.
- The performer won't be interested in completing such tasks because they'll spend a lot of time studying instructions but won't earn much.
- A large pool with 1% of control tasks (good)
-
There are 10,000 tasks in the pool, and 100 of them are control tasks (1%). Each suite contains 10 tasks, and 1 of them is a control task. Hence, a user can complete up to 100 suites.
- A small pool with 1% control tasks (bad)
-
There are 100 tasks in the pool, and 1 of them is a control task (1%). Each suite contains 10 tasks, and 1 of them is a control task. Hence, each user can only complete 1 suite.
- A small pool with 10% control tasks (good)
-
There are 100 tasks in the pool, and 10 of them are control tasks (10%). Each suite contains 10 tasks, and 1 of them is a control task. Hence, each user can complete up to 100 suites
You've stopped the main pool. This could limit the number of performers with access to the pool. Start the training pool again. There will be more performers who can access the tasks.
- The filters you set are too strict. For example, a strong restriction on a certain skill that most users don't have.
- Too many users are banned. Ease the quality control rules.
- To motivate performers, assign a public skill and use dynamic pricing.
- Try to increase the project rating, so that your task is higher in the list of tasks for performers.
- Adjust the quality-speed ratio.
- Set a higher priority for the pool among other project pools.
The settings for quality control rules depend on the type of tasks. General recommendations:
It is better to use one skill in a project. You can choose the way to calculate the skill:
Yes, of course — you can use the same skill for different projects. But most often, a skill is intended for a specific project. If the performer completes a certain task well, this doesn't mean that they will complete other ones successfully. Another disadvantage is that if you filter by skills that were set long ago, you will artificially limit the number of available performers.
Yes. When you copy the filter and quality control settings, the settings you previously added manually are overwritten. You should see a warning about this in the copy settings window.
Yes, the fast response settings specify the time per task suite.
No, this is incorrect. With these settings, each time a rule condition is met, the performer gets skill = 1. To change the skill value in the process of task review, you need a “multi-step” rule, which has multiple identical rules with different values of Total reviewed responses.
Overlap defines how many performers complete the same pool task.
The best overlap is an overlap that provides satisfying quality of results. For most tasks that are not reviewed, overlap from “3” to “5” is enough. If the tasks are simple, overlap of “3” is likely to be enough. For tasks that are reviewed, set overlap to “1”.
Yes. Open edit mode for the pool and set a new overlap value. You don't need to restart the pool. Updating the settings is usually fast, but if there are many tasks, it may take several minutes.
Indeed, this rule is probably too strict. Even the most careful user can make a mistake, so you probably want to relax the rule. Besides the requester-specific bans, we have system processes that ban users who regularly fail captcha checks in Toloka.
Yes, unfortunately, this can happen. This is why we recommend that you offer a training task or exam before the general task. In this case, only those people who showed good performance at the previous stage are selected for the main pool.
If the user already got paid for the tasks, the money can't be refunded to you.
Yes, if they can access both pools, they can do both of them. To restrict access to subsequent tasks for a performer, use the Completed tasks rule and select a ban at the project level.
No. The responses of these performers aren't automatically excluded from the final results file.
But you can do it yourself if you want. When downloading the results, select the option Exclude assignments by banned users to delete the responses of performers who were banned at the moment of downloading. You can also forward all the assignments from banned users to other performers using the Re-completion of assignments from banned users rule.
No, the users are unaware of the ban.
Captcha is usually used in simple projects with automatic acceptance, like classification, categorization, or information search. These are cases where there are few response options and users don't need to upload files or write texts. It helps you filter out bots and sloppy performers.
The percentage of correct responses is based on the total number of captchas processed by the performer within the “range” specified in the Recent control task responses to use field. If the value is empty, the percentage is calculated using all the captchas that are shown for the tasks in the pool which uses the captcha rule.
To perform actions with users (assign a skill or ban them) based on the majority vote, add a relevant rule to the pool.
Don't forget to enable Keep task order in the pool parameters. Majority vote is used in the projects with preset options (radio buttons or checkboxes). This rule won't apply to the text entry or file upload fields.
All responses to the task are taken into account. If one response differs from the majority vote, the whole task is counted as mismatching the responses of other performers.
We recommend adding at least 1% of control tasks in the pool. And for small pools — 5-10%.
Each control task is shown to the performer only once. If you use smart mixing, you determine how many control tasks should be in a suite. If each suite contains one control task, then the maximum number of suites the performer can complete is equal to the number of control tasks in the pool. If you increase the number of control tasks in a suite, the number of suites available to the user decreases by the same number.
There shouldn't be too few pages available. Otherwise:
If there are few control tasks in the open pool, add new control tasks.
In a large pool with few control tasks, a situation might occur when users who have completed a lot of tasks in the project stop getting new task suites. This happens when the user completes all control tasks in the pool.
To filter out performers, use the Control tasks quality control rule. To rank performers by the quality of responses in control tasks, use a skill.
Smart mixing is set up when you upload tasks to the pool. After creating a pool, click Upload and select the method for generating task suites. You can upload them using separate files or one file, arranging them in any order.
The Control tasks rule starts working after the performer completes the number of control tasks you specified. If your pool contains both training and control tasks, you can take into account the responses in both of them (the Number of responses parameter) or only in control tasks (the Number of control responses parameter).
As soon as the needed number of responses is collected, Toloka calculates the percentage of correct and incorrect responses and performs an action (assigns a skill, or blocks the user in the pool or in the project). Then this percentage is updated as the tasks are completed by the performer. The number of the performer's recent responses that's used in the calculation is set in the Recent control task responses to use field. If you leave it empty, all the responses from the performer in the pool are counted.
Technically, if you have only one task in your training pool, you don't have this option. The skill will be either 0 or 100. We recommend that you add several tasks, or at least 2 so that the performer will practice on the first task and will be able to do the second task correctly. In this case, you can admit users to your main pool starting from the skill value of 50.
You can also create a training pool based on the main pool. Assign a skill using the Control tasks rule: in this case, you can admit users with any skill level to your main pool, even if the value is zero. But we don't advise giving tasks to people who failed training.
No, they can't.
The tasks themselves are not exported, only the project configuration and the settings of the selected pool. However, you can download your marked up tasks from the Sandbox pool and import them to the pool you created. To download the control tasks only (if you marked them up in the interface), go to Mark up, then click Control tasks and Download.
This is the total number of responses to the control questions.
You can create a task pool for all your performers and create performer skills in it. In this case, you can open your tasks only to the performers with the necessary skills.
Your training and control tasks have the same project specification. However, you can create a separate project with the tasks and assign a skill based on user responses. Then you can admit performers to the main project based on their skill.
An exam pool contains only control tasks. It's usually small and used for checking how well users learned to do your tasks after they read the instructions and completed the training. Unlike your main pool, you already know the correct responses for every task in this pool. You can set the price to zero.
Based on the results of responses to control tasks, you can assign a skill to the users and then specify it in the main pool as a filter. For example, MySkill = 80 or = Is missing. You don't have to create an exam. For simple tasks, the training pool provides enough practice, but many requesters also use exams.
When you load tasks, use smart mixing. In this case, you'll have infinite overlap in your exam.
However, this poses the risk that you might spend a lot of money on the exam. You might want to open this pool only when the main pool opens, and close it when labeling of the main pool ends.
You can add a training pool to test your performers. Based on the test results, assign skills to the users for the tasks they do best.
Then open your pools only to the users that have a certain skill: use filters for this.
Yes, you can do that. In this case, create the first pool based on the training pool and the exam pool based on your main pool. If a pool contains only control and/or training tasks, the price can be set to zero.
In the exam pool, you can create a skill reflecting the exam result and granting admission to the main pool. For example, if the number of responses is ≥ 10, set the skill value in the <exam skill> as % of correct responses. In your exam pool user requirements, specify: <exam skill> < 80 or = Is missing>. In the main pool, set up a filter: <exam skill> >= 80 and (<main skill> >= 70 or = Is missing). You can choose the skill values depending on how well the performers handle your task.
The Recent control task responses to use field is for the number of recent responses from the performer. If you use non-automatic acceptance for your task, then to set up your intended rule you need to specify 3 in Total reviewed responses.
