Quality control rules

Quality control rules allow you to get more accurate responses and restrict access to tasks to cheating Tolokers. All rules work independently.

List of rules

Quality control presets

Toloka has presets of quality control rules. Currently, there are three of them:
  • Elementary — suitable for requesters who need tasks to be completed by a large number of Tolokers in a short time.
  • Basic — suitable for those who need to balance quality and quantity.
  • Advanced — a set with a large number of rules required for complex tasks.
The table shows the rules included in each of the sets.

Earnings

Skipped assignments

Control tasks

Majority vote

Submitted answers

Fast responses

Elementary + +
Basic + + + +
Advanced + + + + + +

Earnings

Skipped assignments

Control tasks

Majority vote

Submitted answers

Fast responses

Elementary + +
Basic + + + +
Advanced + + + + + +

How to set up quality control

You can configure quality control in the pool and in the customizable project.

Go to pool editing (the Edit button in the upper-right corner of the page) and click Add Quality Control Rule.

You can copy quality control settings from another pool. To do this, click Copy settings from in the Users filter section.

Troubleshooting

Setting up quality control
How do I set quality control in a pool correctly?

The settings for quality control rules depend on the type of tasks. General recommendations:

  • Always use one or more ways to control quality of answers.

  • Counting fast responses makes sense for most tasks.

  • If the user has to choose between options (for example, by selecting checkboxes), check the answers using majority vote or control tasks.

  • If the user has to provide a response as a text or link or upload a photo, the best way to control quality is by reviewing assignments. You can outsource task acceptance to Tolokers. Create a task with a question (for example, “Is this phrase translated correctly?”) and possible responses (for example, “yes”/“no”). Set up overlap and majority vote check.

  • If a task is more like an opinion poll (for example, choosing nice pictures from a set), majority vote is not a good way to control quality. Make control tasks with artificial examples where the choice is evident.

Should I create a skill for every pool?

It is better to use one skill in a project. You can choose the way to calculate the skill:

  • Calculate the skill for each pool separately. The current skill value is the value of the skill in the pool the user completed last. This option is convenient if:

    • The pools are intended for different groups of Tolokers (for example, there are filters by city or country).

    • Pools are started one by one and you don't want to take into account the responses in the previous pools to calculate the skill in the current pool.

    This calculation method is used by default when adding a quality control rule to a pool. For the control tasks block, leave the Recent control task responses to use field empty.

  • Calculate skill based on all tasks in a project This option is good if the pools are small and you don't need to have skill calculated for each pool.

    This option is available only for skills on control tasks. To use it, fill in the Recent control task responses to use field in pool quality control rules.

Can I use a skill beyond a particular pool or project and apply it to other projects as well?

Yes, of course — you can use the same skill for different projects. But most often, a skill is intended for a specific project. If the Toloker completes a certain task well, this doesn't mean that they will complete other ones successfully. Another disadvantage is that if you filter by skills that were set long ago, you will artificially limit the number of available Tolokers.

I set up quality control, then I copied my user requirements. All my quality control settings were deleted and replaced with the copied settings. Is that normal?

Yes. When you copy the filter and quality control settings, the settings you previously added manually are overwritten. You should see a warning about this in the copy settings window.

Is the time specified per task suite in the fast response settings?

Yes, the fast response settings specify the time per task suite.

Have I understood correctly that if I use set the the skill value = 1 with the percentage of accepted responses >= 75 and 10 recent values to use, for