Dynamic overlap

Theory

Dynamic overlap lets you change overlap depending on:

  • How well the Tolokers do on the task.
  • How well Tolokers' task responses match each other.

First the task is distributed with minimal overlap. As Tolokers complete the task, their responses are assigned confidence. If it is less than specified in the confidence in aggregated response field, overlap will be increased until confidence reaches the set value, or overlap reaches the maximum.

Dynamic overlap uses 5 main parameters:

  1. Overlap is the minimum number of Tolokers who should complete each task in the pool.

  2. Maximum overlap is the maximum number of Tolokers who should complete each task in the pool.

  3. Confidence in aggregated response is the percentage of confidence required to stop showing the task to Tolokers.

    When confidence reaches the required level, the system considers the task response correct and stops collecting other Tolokers' responses.

  4. Skill reflects how much you trust the Toloker's responses.

  5. Output fields for aggregation are output data fields that are used to calculate confidence in the response.

How does it work?

Practice

Before setting up dynamic overlap, you need to prepare the output data fields.

You can use the following fields:
You can't use the following fields:

Setting up overlap

  1. Create a pool and fill in the parameters.

  2. Set the minimum overlap in the Overlap field.

  3. In the advanced settings, enable the Use dynamic overlap option.

  4. Set Max overlap.

  5. Specify Confidence of aggregated response. The higher the value, the more matching Tolokers' responses are needed for the aggregated response. Set this value to at least 70.

  6. Choose Skill for users' scoring.

  7. Check Output fields for answers agreement.

  8. Save the pool.

  9. Upload tasks via smart mixing.

  10. After the pool is completed, start Aggregation of results by skill.

Note

Approximate budget in the pool statistics is calculated based on the minimal overlap.

Example

Tips and recommendations

  • Don't use the Recompletion of assignments from banned users quality control rule together with dynamic overlap. This increases the maximum overlap value.

  • Set confidence in the aggregated response at between 70 and 95 percent.

    • When the value is < 70%, the average accuracy of the aggregated response isn't sufficient to trust the Tolokers' responses.

    • When the value is > 90%, the average accuracy of the aggregated response doesn't increase, and the labelling costs increase.

    Optimal confidence is at 80% for the best “price/quality” ratio.

  • Upload tasks via smart mixing. Otherwise, the tasks are assigned with minimum overlap.

    This is necessary for overlap to increase because control and training tasks uploaded via smart mixing have infinite overlap.

  • Use a skill calculated as the percentage of correct responses to control tasks. This will give you the most accurate results.

  • If you edit a required field, the changes apply only to new pools. Existing pools will continue using the previous version of the project.

What's next

See also

For developers

Troubleshooting

Can I change overlap after the pool is started?
With dynamic overlap, is it possible that the pool will close before the tasks for minimal overlap run out? The overlap increased, but the pool is closed, and I need to start it manually.
Contact support

Last updated: February 15, 2023

Introduction
Getting started
Important tips
Useful recommendations
Working with Toloka
Projects
Pools
Tasks
Results
Project analysis
Toloka settings
Task interface
Template Builder
HTML/CSS/JS editor
Help and support
FAQTroubleshootingSupportGlossary