Majority vote is a quality control method based on matching responses from the majority of Tolokers who complete the same task. For example, if a task has an overlap of “5” and three Tolokers selected the same answer, this is considered indirect confirmation of the correct response.
Based on the percentage of correct responses, you can change the value of the Toloker's skill or block their access to tasks.
Let's say we set up this rule in a pool with
It is activated when at least three of the tasks completed by the Toloker reach full overlap. When responses have been received from all five task Tolokers, the system determines which response was selected by the majority. If the Toloker's responses to two out of three tasks differ from the responses of the majority, the Toloker is banned from the project for 10 days. To collect all the responses needed to trigger the rule, enable Keep task order in the pool settings.
The number of tasks per suite doesn't affect how the rule works. Responses are evaluated separately for each task. Only the speed of collecting results can be affected, because the more tasks you put on each page, the longer it takes to do them.
When to use
- Use the Majority vote rule if:
- You can't regularly create and add control tasks.
- You want to make an additional review of the Toloker's responses.
- Don't use it if:
- You have a lot of response options.
- Tolokers need to attach a file to their assignment.
- Tolokers need to transcribe text.
- Tolokers need to select objects in a photo.
- Tasks don't have a correct or incorrect response. For example: “Which image do you like best?” or “Choose the page design option that you like best”.
- Don't use too many response options or too many questions in the task.
The more options you have, the lower the probability that majority vote will be calculated correctly.
To calculate the majority vote for the task, all the responses to all the questions inside the task must match. If a task has multiple questions or the questions have more than just a few response options, it will be unlikely to have enough exactly matching task responses from different Tolokers in order to calculate the skill. As a result, the task won't be used for skill calculation. If the majority vote hasn't been calculated for the task, the Toloker isn't penalized for mismatching the majority vote.
Toloker 1 Toloker 2 Toloker 3 Toloker 4 Toloker 5 Task 1 OK BAD OK BAD OK Task 2 BAD OK OK BAD 404 Task 3 404 OK 404 OK OK Task 4 OK BAD OK 404 404 Task 5 OK BAD OK 404 OK Toloker 1 Toloker 2 Toloker 3 Toloker 4 Toloker 5 Task 1 OK BAD OK BAD OK Task 2 BAD OK OK BAD 404 Task 3 404 OK 404 OK OK Task 4 OK BAD OK