What is overlap?

Overlap is the number of performers who should complete each task in the pool.

You can set overlap in the pool settings:

Tips and recommendations

Task type

Recommended overlap

Most tasks have automatic acceptance

from 3 to 5
Simple task 3
Reviewed task 1

Task type

Recommended overlap

Most tasks have automatic acceptance

from 3 to 5
Simple task 3
Reviewed task 1

You can change overlap after the pool is started. To do this, open the pool editing mode and set the new overlap value. You don't need to restart the pool. Updating the settings is usually fast, but if there are many tasks, it may take several minutes.

Advanced overlap types
  • Dynamic overlap lets you change overlap depending on:
    • How well the performers do on the task.
    • How well performers' responses match each other.

    This type of overlap helps you save money without reducing the quality of data labeling.

  • Selective majority vote verification allows you to set up majority vote check for only a portion of tasks.

    This type of overlap helps you save money and speed up pool completion.

Troubleshooting

Can I do it like this: set a basic overlap of 2 users, then, if both performers select the same response, close the pool, but if they give different responses, show the task to one more user?

Yes, you can do that. Set up dynamic overlap (incremental relabeling, IRL).

Is there a cross-check feature for tasks?

You can use overlap to let multiple performers do the same task. The overlap value is set up in the pool settings.

Why is the maximum number of submitted assignments in the progress bar less than the total number of uploaded tasks?

The progress bar shows the number of task suites including the overlap. If the overlap is greater than one, the number of task suites is different from the total number of tasks.

How does counting work if I set overlap = 3 in the pool and response threshold = 3 in the majority vote?

In this case, if you don't have 3 identical responses for your task (response threshold), no user would be considered a good or poor performer, because the system can't see which of the users made an error.

But if you set response threshold = 2 with overlap = 3, then two users with the same responses are considered good performers, but the third user, who gives a different response, is a poor performer.