Speed/quality balance
By default, your tasks are available to all Tolokers. If you set filters, they are available only to those who match them.
When there are many Tolokers, you can get results quickly, but the quality of the Tolokers' responses may be poor.
If you want to improve speed or quality of labeling, use the Speed/quality balance pool setting.
It allows you to select Tolokers based on a personalized quality forecast:
-
TOP % — allows you to set the percentage of top-rated Tolokers who will have access to your task.
Example
Only 40% of top-rated Tolokers will have access to your task.
-
Online — allows you to set the maximum number of top-rated Tolokers who will have access to your task.
Example
Only 3000 of top-rated Tolokers will have access to your task.
How does it work?
A personalized quality forecast is based on a large amount of data about user behavior in the system, how other Tolokers completed your task, and the task itself. Based on this data, we predict in real time how well each particular user will handle your task. We select top-rated Tolokers by comparing quality forecasts for all Tolokers.
Recommendations
-
Make sure that the total number of Tolokers exceeds the one required to complete your task several times. Otherwise, your task may not be completed due to lack of Tolokers.
-
If you don't know what settings to choose, make the task available only to top-rated Tolokers. For example, set 20% in the TOP % setting. If the tasks completion speed is insufficient, increase the value to get results faster.
What's next
-
Add tasks to the pool.
-
Learn more about how to set up a pool:
- Setting up pricing.
- Dynamic overlap.
- Selective majority vote control.
- Filters.
- Setting up quality control.
- Reviewed assignments.
See also
For developers
Troubleshooting
Possible reasons:
-
You've stopped the training pool. This could limit the number of Tolokers with access to the pool. Start the training pool again. There will be more Tolokers who can access the tasks.
-
The filters you set are too strict. For example, a strong restriction on a certain skill that most Tolokers don't have.
-
Too many Tolokers are banned. Ease the quality control rules.
-
To motivate Tolokers, assign a public skill and use dynamic pricing.
-
Try to increase the project rating, so that your task is higher in the list of tasks for Tolokers.
-
Adjust the quality-speed ratio.
-
Set a higher priority for the pool among other project pools.