Majority vote

Majority vote is a quality control method based on matching responses from the majority of Tolokers who complete the same task. For example, if a task has an overlap of “5” and three Tolokers selected the same answer, this is considered indirect confirmation of the correct response.

Note. It only takes into account regular tasks, ignoring the responses to control and training tasks.

Based on the percentage of correct responses, you can change the value of the Toloker's skill or block their access to tasks.

The rule is activated when the task overlap is complete.

Let's say we set up this rule in a pool with overlap=5:

It is activated when at least three of the tasks completed by the Toloker reach full overlap. When responses have been received from all five task Tolokers, the system determines which response was selected by the majority. If the Toloker's responses to two out of three tasks differ from the responses of the majority, the Toloker is banned from the project for 10 days. To collect all the responses needed to trigger the rule, enable Keep task order in the pool settings.

The number of tasks per suite doesn't affect how the rule works. Responses are evaluated separately for each task. Only the speed of collecting results can be affected, because the more tasks you put on each page, the longer it takes to do them.

When to use

Use the Majority vote rule if:
  • You can't regularly create and add control tasks.
  • You want to make an additional review of the Toloker's responses.
Don't use it if:
  • You have a lot of response options.
  • Tolokers need to attach a file to their assignment.
  • Tolokers need to transcribe text.
  • Tolokers need to select objects in a photo.
  • Tasks don't have a correct or incorrect response. For example: “Which image do you like best?” or “Choose the page design option that you like best”.

Usage recommendations

Don't use too many response options or too many questions in the task.

The more options you have, the lower the probability that majority vote will be calculated correctly.

To calculate the majority vote for the task, all the responses to all the questions inside the task must match. If a task has multiple questions or the questions have more than just a few response options, it will be unlikely to have enough exactly matching task responses from different Tolokers in order to calculate the skill. As a result, the task won't be used for skill calculation. If the majority vote hasn't been calculated for the task, the Toloker isn't penalized for mismatching the majority vote.

Toloker 1 Toloker 2 Toloker 3 Toloker 4 Toloker 5
Task 2 BAD OK OK BAD 404
Task 3 404 OK 404 OK OK
Task 4 OK BAD OK 404 404
Task 5 OK BAD OK 404 OK
Toloker 1 Toloker 2 Toloker 3 Toloker 4 Toloker 5
Task 2 BAD OK OK BAD 404
Task 3 404 OK 404 OK OK
Task 4 OK BAD OK