Majority vote

Majority vote is a quality control method based on matching task responses from the majority of Tolokers who complete the same task. For example, if a task has an overlap of “5” and three Tolokers selected the same answer, this is considered indirect confirmation of the correct task response.

Note

It only takes into account regular tasks, ignoring the responses to control and training tasks.

Based on the percentage of correct task responses, you can change the value of the Toloker's skill or block their access to tasks.

The rule is activated when the task overlap is complete.

Example

When to use

Use the Majority vote rule if:

  • You can't regularly create and add control tasks.
  • You want to make an additional review of the Toloker's task responses.

Don't use it if:

  • You have a lot of task response options.
  • Tolokers need to attach a file to their assignment.
  • Tolokers need to transcribe text.
  • Tolokers need to select objects in a photo.
  • Tasks don't have a correct or incorrect task response. For example: “Which image do you like best?” or “Choose the page design option that you like best”.

Usage recommendations

  • Don't use too many task response options or too many questions in the task.

    The more options you have, the lower the probability that majority vote will be calculated correctly.

    To calculate the majority vote for the task, all the task responses to all the questions inside the task must match. If a task has multiple questions or the questions have more than just a few task response options, it will be unlikely to have enough exactly matching task responses from different Tolokers in order to calculate the skill. As a result, the task won't be used for skill calculation. If the majority vote hasn't been calculated for the task, the Toloker isn't penalized for mismatching the majority vote.

    Example

    Let's say the task Toloker has to select at least three matching categories out of twenty or report that the image is not displayed. Since there can be multiple combinations of categories, the rule calculation will mostly include tasks where all the Tolokers responded Not displayed. Those might be either correct or “fraudulent” task responses from Tolokers who just clicked through the tasks in a hurry. In any case, it would be unreliable to base the skill on these task responses.

    Try using decomposition or control tasks. This way you can better assess the quality of your Tolokers.

  • Don't set the threshold Accept as majority below 50% of the overlap or equal to the overlap.

    If the task response threshold is equal to the overlap, the rule will only use the tasks where all the Tolokers gave the same task responses, ignoring tasks with mismatching votes.

    Example

  • Don't use the majority vote if:

    • You have a lot of task response options.

    • Tolokers need to attach a file to their assignment.

    • Tolokers need to transcribe text or select objects in a photo, or other tasks where the Tolokers can't possibly provide the same task responses.

Rule settings

Warning

In this rule, all fields are required except Recent tasks to use. If you don't fill in at least one of them, you won't be able to save the rule.

FieldOverview
Accept as majorityThe number of matching task responses that is considered the "majority vote" (for example, 3).
Recent tasks to useHow many recent task responses from the Toloker to use.

If this field is not filled in, the calculation includes only task responses from the pool to which the rule is applied.

If the field is filled in, the corresponding number of task responses is used. The rule takes into account task responses from both the current pool and other pools where this field is filled in.

Learn more about how this field works.
If

A condition for performing the action in the then field:

  • number of task responses — The number of completed tasks.

  • correct task responses (%) — The percentage of correct task responses, meaning task responses that matched the majority opinion (from 0 to 100).

  • incorrect task responses (%) — The percentage of incorrect task responses, meaning task responses that didn't match the majority opinion (from 0 to 100).

To add multiple conditions, click .

then

Action to perform for the If condition:

  • ban — Block access to the project or all of the requester's projects for the specified number of days. Only the requester can view the reason.

    If access to tasks is blocked temporarily (for example, for 7 days), the history of the Toloker's task responses is not saved after the ban is lifted. The skill level is calculated based on the new task responses.

  • assign skill from the field — Save the percentage of the Toloker's correct task responses in tasks as a skill value.

  • assign skill value — Assign a fixed value to the skill.

  • suspend — Suspend the Toloker's access to the pool for the specified number of days. Only the requester can view the reason.

  • accept all assignments from this Toloker in the pool — Requires the manual review option to be set.

    Useful if the Toloker completes most tasks well. Example: The Toloker completed more than 80% of the tasks correctly and you are satisfied with this result. The rule will work automatically and accept all task responses in the pool.

Learn more about how this field works.

Examples of rules

Examples are provided for simple classification. There are 10 tasks per suite.

Warning

The assignments submitted by banned Tolokers will be taken into account if they are not rejected using manual review. They can be reassigned by setting up the Recompletion of assignments from banned users rule.

Calculating a skill and banning for incorrect responses

Correct settings
Incorrect settings
Alternative settings

Both rules work independently:

  1. If the Toloker gives at least 3 responses to the tasks, the percentage of correct answers is written as the skill value.

  2. If the Toloker gives at least 3 answers to tasks and the percentage of correct answers is less than 65%, they are banned on the project.

The calculation uses up to 10 Toloker's responses to the project tasks.

Setting the skill

Correct settings
Incorrect settings

If the Toloker completes 3 tasks, the skill is set to the percentage of correct answers.

Use the skill value to set access to other pools with filters.

Example of filter settings

Blocking for incorrect task responses

Correct settings
Incorrect settings

If the percentage of correct task responses is less than 40%, the Toloker is blocked on the project for 30 days.

See also

For developers

Troubleshooting

Should I create a skill for every pool?
Can I use a skill beyond a particular pool or project and apply it to other projects as well?
What output format do I use for the review results to filter out mismatching Tolokers based on the “Majority vote”?
The pool has an overlap and majority vote set up, but some fraudulent Toloker opens the task suites, does nothing, and submits empty assignments. Could this cheater get more tasks from the pool before the results of other Tolokers are known? Could a Toloker quickly click through a lot of task suites before the majority vote is accumulated to ban the cheater?
My task uses a form with multiple fields. When there is an overlap and “Majority vote” is used for quality control, is each field taken into account, or if one field mismatches the majority vote, are the task results considered incorrect?
Contact support

Last updated: July 26, 2023

Introduction
Getting started
Important tips
Useful recommendations
Working with Toloka
Projects
Pools
Tasks
Results
Project analysis
Toloka settings
Task interface
Template Builder
HTML/CSS/JS editor
Help and support
FAQTroubleshootingSupportGlossary