Selective majority vote check

Before you start

To set up selective majority vote check, you need a project, a pool, and a file with tasks.

Theory

This function allows you to set up majority vote check for only a portion of tasks.

Depending on how many tasks the performer completed in N days, overlap for certain tasks will be higher or lower than in the pool settings.

How it works
Reducing overlap

The pool's overlap is set to 4. To speed up pool completion and reduce data labeling costs, we'll reduce overlap for every third, fifth, and tenth task.

Performer A completed 15 tasks in the last 10 days. Performer B completed 30 tasks. Performer C completed 60 tasks.

For performer A, overlap will be set to 3 for every third task; for performer B — for every fifth task; for performer C — for every tenth task.

This way we can reduce overlap for more experienced performers.

Increasing overlap

The pool's overlap is set to 2. This project has experienced performers who perform tasks well. We want to increase overlap for new performers and check the experienced performers' responses from time to time. To do this, we will increase overlap for every first, third, fifth, and tenth task.

Performer A completed 5 tasks in the last 10 days. Performer B completed 15 tasks. Performer C completed 25 tasks. Performer E completed 40 tasks.

For performer A, overlap will be set to 5 for every task; for performer B — for every third task; for performer C — for every fifth task; for performer E — for every tenth task.

This way we will increase overlap for inexperienced users and save money on the responses of experienced performers.

When to use
Use selective majority vote check if:
  • There are few or no control tasks in the project.

  • You have experienced performers whom you trust.

Practice

To add selective majority vote check:

  1. Upload tasks to the pool using smart mixing.

  2. Switch to pool editing mode.

  3. Under Smart mixing settings, click .

  4. Specify how many days the calculation should take into account, performers, and overlap for majority vote tasks.

  5. Set the intervals of completed tasks and the number of review tasks for each interval.

Example

Tips and recommendations

  • Don't use selective majority vote check if:
    • There are many questions and many response options in the task.

    • Users need to attach a file to their assignment.

    • Users need to transcribe text or select objects in a photo, or other tasks where the performers can't possibly provide the same responses.

  • Use selective majority vote check together with the fast responses and skipped assignments quality control rules. This way, you will eliminate dishonest performers who skip tasks or complete them too quickly, which directly affects the number of completed tasks and overlap.