Setting up quality control
Quality control lets you get more accurate responses and restrict access to tasks for cheating Tolokers. Quality control consists of rules. All rules work independently.
When you clone a project, its quality control settings aren't transferred.
- Open the project page.
- Go to the Quality control tab.
- Click Set quality control.
- Click Add Quality Control Rule.
- In the list that appears, select the appropriate rules. If you aren't sure what quality control rules you need, select a ready-made protection set of rules with default settings.
- Make settings for the rules you added. Below is a list of rules with links to detailed information about the rule settings.
List of rules
- To keep track of how often Tolokers make mistakes:
- To protect your project from robots and cheaters:
- Captcha: Show a captcha from time to time to make sure tasks aren't completed by robots.
Always use one or more ways to control quality of answers.
Counting fast responses makes sense for most tasks.
If the Toloker has to provide a response as a text or link or upload a photo, the best way to control quality is by reviewing assignments. You can outsource task acceptance to Tolokers. Create a task with a question (for example, “Is this phrase translated correctly?”) and possible responses (for example, “yes”/“no”). Set up overlap and majority vote check.
- You won't be able to correctly evaluate the quality of the Toloker's responses.
- The Toloker won't be interested in completing such tasks because they'll spend a lot of time studying instructions but won't earn much.
- A large pool with 1% of control tasks (good)
There are 10,000 tasks in the pool, and 100 of them are control tasks (1%). Each suite contains 10 tasks, and 1 of them is a control task. Hence, a user can complete up to 100 suites.
- A small pool with 1% control tasks (bad)
There are 100 tasks in the pool, and 1 of them is a control task (1%). Each suite contains 10 tasks, and 1 of them is a control task. Hence, each user can only complete 1 suite.
- A small pool with 10% control tasks (good)
There are 100 tasks in the pool, and 10 of them are control tasks (10%). Each suite contains 10 tasks, and 1 of them is a control task. Hence, each user can complete up to 100 suites
The settings for quality control rules depend on the type of tasks. General recommendations:
We recommend adding at least 1% of control tasks in the pool. And for small pools — 5-10%.
Each control task is shown to the Toloker only once. If you use smart mixing, you determine how many control tasks should be in a suite. If each suite contains one control task, then the maximum number of suites the Toloker can complete is equal to the number of control tasks in the pool. If you increase the number of control tasks in a suite, the number of suites available to the Toloker decreases by the same number.
There shouldn't be too few pages available. Otherwise:
If there are few control tasks in the open pool, add new control tasks.
In a large pool with few control tasks, a situation might occur when users who have completed a lot of tasks in the project stop getting new task suites. This happens when the Toloker completes all control tasks in the pool.
The Control tasks rule starts working after the Toloker completes the number of control tasks you specified. If your pool contains both training and control tasks, you can take into account the responses in both of them (the Number of responses parameter) or only in control tasks (the Number of control responses parameter).
As soon as the needed number of responses is collected, Toloka calculates the percentage of correct and incorrect responses and performs an action (assigns a skill, or blocks the Toloker in the pool or in the project). Then this percentage is updated as the tasks are completed by the Tolo