# Majority vote

Majority vote is a quality control method based on matching responses from the majority of Tolokers who complete the same task. For example, if a task has an overlap of “5” and three Tolokers selected the same answer, this is considered indirect confirmation of the correct response.

Note

It only takes into account regular tasks, ignoring the responses to control and training tasks.

Based on the percentage of correct responses, you can change the value of the Toloker's skill or block their access to tasks.

The rule is activated when the task overlap is complete.

Example

Let's say we set up this rule in a pool with overlap=5:

It is activated when at least three of the tasks completed by the Toloker reach full overlap. When responses have been received from all five task Tolokers, the system determines which response was selected by the majority. If the Toloker's responses to two out of three tasks differ from the responses of the majority, the Toloker is banned from the project for 10 days. To collect all the responses needed to trigger the rule, enable Keep task order in the pool settings.

The number of tasks per suite doesn't affect how the rule works. Responses are evaluated separately for each task. Only the speed of collecting results can be affected, because the more tasks you put on each page, the longer it takes to do them.

## When to use

#### Use the Majority vote rule if:

• You want to make an additional review of the Toloker's responses.

#### Don't use it if:

• You have a lot of response options.
• Tolokers need to attach a file to their assignment.
• Tolokers need to transcribe text.
• Tolokers need to select objects in a photo.
• Tasks don't have a correct or incorrect response. For example: “Which image do you like best?” or “Choose the page design option that you like best”.

## Usage recommendations

#### Don't use too many response options or too many questions in the task.

The more options you have, the lower the probability that majority vote will be calculated correctly.

To calculate the majority vote for the task, all the responses to all the questions inside the task must match. If a task has multiple questions or the questions have more than just a few response options, it will be unlikely to have enough exactly matching task responses from different Tolokers in order to calculate the skill. As a result, the task won't be used for skill calculation. If the majority vote hasn't been calculated for the task, the Toloker isn't penalized for mismatching the majority vote.

Example
Toloker 1 Toloker 2 Toloker 3 Toloker 4 Toloker 5
Task 3 404 OK 404 OK OK

The requester has five tasks, each completed by five Tolokers. Each task has one output field with three response options.Overlap = 3. In this case, tasks 1, 3, and 5 match the threshold, but the other tasks are not included in the majority vote calculation.

The Tolokers will be assigned the following skills based on majority vote:

 Toloker 1 (1+0+1)/3=66% correct Toloker 2 (0+1+0)/3=33% correct Toloker 3 (1+0+1)/3= 66% correct Toloker 4 (0+1+0)/3=33% correct Toloker 5 (1+1+1)/3=100% correct

Let's say the task Toloker has to select at least three matching categories out of twenty or report that the image is not displayed. Since there can be multiple combinations of categories, the rule calculation will mostly include tasks where all the Tolokers responded Not displayed. Those might be either correct or “fraudulent” responses from Tolokers who just clicked through the tasks in a hurry. In any case, it would be unreliable to base the skill on these responses.

Try using decomposition or control tasks. This way you can better assess the quality of your Tolokers.

#### Don't set the threshold Accept as majority below 50% of the overlap or equal to the overlap.

If the response threshold is equal to the overlap, the rule will only use the tasks where all the Tolokers gave the same responses, ignoring tasks with mismatching votes.

Example

Let's say the pool has Overlap = 5 and Accept as majority = 2. The Tolokers completed the task.

Toloker 1 OK
Toloker 2 OK
Toloker 5 404

In this case, Tolokers 1, 2, 3, and 4 responded correctly, but Toloker 5 failed. However, you don't have any way to know which response is correct.

#### Don't use the majority vote if:

• You have a lot of response options.
• Tolokers need to attach a file to their assignment.
• Tolokers need to transcribe text or select objects in a photo, or other tasks where the Tolokers can't possibly provide the same responses.

## Rule settings

Warning

In this rule, all fields are required except Recent tasks to use. If you don't fill in at least one of them, you won't be able to save the rule.

## Examples of rules

Examples are provided for simple classification. There are 10 tasks per suite.

Warning

The assignments submitted by banned Tolokers will be taken into account if they are not rejected using manual review. They can be reassigned by setting up the Recompletion of assignments from banned users rule.

#### Calculating a skill and banning for incorrect responses

Both rules work independently:

1. If the Toloker gives at least 3 responses to the tasks, the percentage of correct answers is written as the skill value.
2. If the Toloker gives at least 3 answers to tasks and the percentage of correct answers is less than 65%, they are banned on the project.

The calculation uses up to 10 Toloker's responses to the project tasks.

The Toloker is blocked after the first incorrect response to the first, second or third task. However, the skill isn't set. Since the ban reason is not specified, there is no way to find out why the Toloker is banned.

All rules are applied independently:

1. If the Toloker gives at least 3 responses to the tasks, the percentage of correct answers is written as the skill value.

2. If the Toloker gives 2 incorrect responses to 3 tasks, they are blocked in the pool for 10 days.

3. If the Toloker gives 2 incorrect responses to 4 tasks, they are blocked in the pool for 10 days.

4. If the Toloker gives 5 or more responses to tasks and the percentage of correct responses is less than 80%, they are blocked in the pool for 10 days.

A set of rules like this prevents Tolokers from being banned for one incorrect response and lets you maintain high accuracy.

#### Setting the skill

If the Toloker completes 3 tasks, the skill is set to the percentage of correct answers.

Use the skill value to set access to other pools with filters.

Example of filter settings

This rule will never take effect because the number of responses counted (Recent values to use) is less than the number of responses in the rule (number of responses).

#### Blocking for incorrect responses

If the percentage of correct responses is less than 40%, the Toloker is blocked on the project for 30 days.

If the percentage of correct responses is less than 40%, the Toloker is blocked on the project for 30 days. The rule will be applied once, after the fifth response.

## Troubleshooting

Should I create a skill for every pool?

It is better to use one skill in a project. You can choose the way to calculate the skill:

• Calculate the skill for each pool separately. The current skill value is the value of the skill in the pool the Toloker completed last. This option is convenient if:

• The pools are intended for different groups of Tolokers (for example, there are filters by city or country).

• Pools are started one by one and you don't want to take into account the responses in the previous pools to calculate the skill in the current pool.

This calculation method is used by default when adding a quality control rule to a pool. For the control tasks block, leave the Recent control task responses to use field empty.

• Calculate skill based on all tasks in a project This option is good if the pools are small and you don't need to have skill calculated for each pool.

This option is available only for skills on control tasks. To use it, fill in the Recent control task responses to use field in pool quality control rules.

Can I use a skill beyond a particular pool or project and apply it to other projects as well?

Yes, of course — you can use the same skill for different projects. But most often, a skill is intended for a specific project. If the Toloker completes a certain task well, this doesn't mean that they will complete other ones successfully. Another disadvantage is that if you filter by skills that were set long ago, you will artificially limit the number of available Tolokers.

What output format do I use for the review results to filter out mismatching users based on the “Majority vote”?

To perform actions with users (assign a skill or ban them) based on the majority vote, add a relevant rule to the pool.

Don't forget to enable Keep task order in the pool parameters. Majority vote is used in the projects with preset options (radio buttons or checkboxes). This rule won't apply to the text entry or file upload fields.

The pool has an overlap and majority vote set up, but some fraudulent user opens the task suites, does nothing, and submits empty assignments. Could this cheater get more tasks from the pool before the results of other Tolokers are known? Could a user quickly click through a lot of task suites before the majority vote is accumulated to ban the cheater?

Yes, unfortunately, this can happen. This is why we recommend that you offer a training task or exam before the general task. In this case, only those people who showed good performance at the previous stage are selected for the main pool.

My task uses a form with multiple fields. When there is an overlap and “Majority vote” is used for quality control, is each field taken into account, or if one field mismatches the majority vote, are the task results considered incorrect?

All responses to the task are taken into account. If one response differs from the majority vote, the whole task is counted as mismatching the responses of other Tolokers.