Majority vote is a quality control method based on matching task responses from the majority of Tolokers who complete the same task. For example, if a task has an overlap of “5” and three Tolokers selected the same answer, this is considered indirect confirmation of the correct task response.
Based on the percentage of correct task responses, you can change the value of the Toloker's skill or block their access to tasks.
The rule is activated when the task overlap is complete.
Use the Majority vote rule if:
Don't use it if:
Don't use too many task response options or too many questions in the task.
The more options you have, the lower the probability that majority vote will be calculated correctly.
To calculate the majority vote for the task, all the task responses to all the questions inside the task must match. If a task has multiple questions or the questions have more than just a few task response options, it will be unlikely to have enough exactly matching task responses from different Tolokers in order to calculate the skill. As a result, the task won't be used for skill calculation. If the majority vote hasn't been calculated for the task, the Toloker isn't penalized for mismatching the majority vote.
Let's say the task Toloker has to select at least three matching categories out of twenty or report that the image is not displayed. Since there can be multiple combinations of categories, the rule calculation will mostly include tasks where all the Tolokers responded Not displayed. Those might be either correct or “fraudulent” task responses from Tolokers who just clicked through the tasks in a hurry. In any case, it would be unreliable to base the skill on these task responses.
Don't set the threshold Accept as majority below 50% of the overlap or equal to the overlap.
If the task response threshold is equal to the overlap, the rule will only use the tasks where all the Tolokers gave the same task responses, ignoring tasks with mismatching votes.
Don't use the majority vote if:
You have a lot of task response options.
Tolokers need to attach a file to their assignment.
Tolokers need to transcribe text or select objects in a photo, or other tasks where the Tolokers can't possibly provide the same task responses.
In this rule, all fields are required except Recent tasks to use. If you don't fill in at least one of them, you won't be able to save the rule.
|Accept as majority||The number of matching task responses that is considered the "majority vote" (for example, 3).|
|Recent tasks to use||How many recent task responses from the Toloker to use.|
If this field is not filled in, the calculation includes only task responses from the pool to which the rule is applied.
If the field is filled in, the corresponding number of task responses is used. The rule takes into account task responses from both the current pool and other pools where this field is filled in.
Learn more about how this field works.
A condition for performing the action in the then field:
To add multiple conditions, click .
Action to perform for the If condition:
Learn more about how this field works.
Examples are provided for simple classification. There are 10 tasks per suite.
The assignments submitted by banned Tolokers will be taken into account if they are not rejected using manual review. They can be reassigned by setting up the Recompletion of assignments from banned users rule.
Both rules work independently:
If the Toloker gives at least 3 responses to the tasks, the percentage of correct answers is written as the skill value.
If the Toloker gives at least 3 answers to tasks and the percentage of correct answers is less than 65%, they are banned on the project.
The calculation uses up to 10 Toloker's responses to the project tasks.
If the Toloker completes 3 tasks, the skill is set to the percentage of correct answers.
Use the skill value to set access to other pools with filters.
If the percentage of correct task responses is less than 40%, the Toloker is blocked on the project for 30 days.
Last updated: July 26, 2023