Majority vote
Majority vote is a quality control method based on matching responses from the majority of Tolokers who complete the same task. For example, if a task has an overlap of “5” and three Tolokers selected the same answer, this is considered indirect confirmation of the correct response.
Note
It only takes into account regular tasks, ignoring the responses to control and training tasks.
Based on the percentage of correct responses, you can change the value of the Toloker's skill or block their access to tasks.
The rule is activated when the task overlap is complete.
Let's say we set up this rule in a pool with overlap=5:
It is activated when at least three of the tasks completed by the Toloker reach full overlap. When responses have been received from all five task Tolokers, the system determines which response was selected by the majority. If the Toloker's responses to two out of three tasks differ from the responses of the majority, the Toloker is banned from the project for 10 days. To collect all the responses needed to trigger the rule, enable Keep task order in the pool settings.
The number of tasks per suite doesn't affect how the rule works. Responses are evaluated separately for each task. Only the speed of collecting results can be affected, because the more tasks you put on each page, the longer it takes to do them.
When to use
Use the Majority vote rule if:
- You can't regularly create and add control tasks.
- You want to make an additional review of the Toloker's responses.
Don't use it if:
- You have a lot of response options.
- Tolokers need to attach a file to their assignment.
- Tolokers need to transcribe text.
- Tolokers need to select objects in a photo.
- Tasks don't have a correct or incorrect response. For example: “Which image do you like best?” or “Choose the page design option that you like best”.
Usage recommendations
Don't use too many response options or too many questions in the task.
The more options you have, the lower the probability that majority vote will be calculated correctly.
To calculate the majority vote for the task, all the responses to all the questions inside the task must match. If a task has multiple questions or the questions have more than just a few response options, it will be unlikely to have enough exactly matching task responses from different Tolokers in order to calculate the skill. As a result, the task won't be used for skill calculation. If the majority vote hasn't been calculated for the task, the Toloker isn't penalized for mismatching the majority vote.
| Toloker 1 | Toloker 2 | Toloker 3 | Toloker 4 | Toloker 5 | |
|---|---|---|---|---|---|
| Task 1 | OK | BAD | OK | BAD | OK |
| Task 2 | BAD | OK | OK | BAD | 404 |
| Task 3 | 404 | OK | 404 | OK | OK |
| Task 4 | OK | BAD | OK | 404 | 404 |
| Task 5 | OK | BAD | OK | 404 | OK |
The requester has five tasks, each completed by five Tolokers. Each task has one output field with three response options.Overlap = 3. In this case, tasks 1, 3, and 5 match the threshold, but the other tasks are not included in the majority vote calculation.
The Tolokers will be assigned the following skills based on majority vote:
|
Toloker 1 |
(1+0+1)/3=66% correct |
|
Toloker 2 |
(0+1+0)/3=33% correct |
|
Toloker 3 |
(1+0+1)/3= 66% correct |
|
Toloker 4 |
(0+1+0)/3=33% correct |
|
Toloker 5 |
(1+1+1)/3=100% correct |
Let's say the task Toloker has to select at least three matching categories out of twenty or report that the image is not displayed. Since there can be multiple combinations of categories, the rule calculation will mostly include tasks where all the Tolokers responded Not displayed. Those might be either correct or “fraudulent” responses from Tolokers who just clicked through the tasks in a hurry. In any case, it would be unreliable to base the skill on these responses.
Try using decomposition or control tasks. This way you can better assess the quality of your Tolokers.
Don't set the threshold Accept as majority below 50% of the overlap or equal to the overlap.
If the response threshold is equal to the overlap, the rule will only use the tasks where all the Tolokers gave the same responses, ignoring tasks with mismatching votes.
Let's say the pool has Overlap = 5 and Accept as majority = 2. The Tolokers completed the task.
| Task response | |
|---|---|
| Toloker 1 | OK |
| Toloker 2 | OK |
| Toloker 3 | BAD |
| Toloker 4 | BAD |
| Toloker 5 | 404 |
In this case, Tolokers 1, 2, 3, and 4 responded correctly, but Toloker 5 failed. However, you don't have any way to know which response is correct.
Don't use the majority vote if:
- You have a lot of response options.
- Tolokers need to attach a file to their assignment.
- Tolokers need to transcribe text or select objects in a photo, or other tasks where the Tolokers can't possibly provide the same responses.
Rule settings
Warning
In this rule, all fields are required except Recent tasks to use. If you don't fill in at least one of them, you won't be able to save the rule.
|
Field |
Overview |
|
Accept as majority |
The number of matching responses that is considered the "majority vote" (for example, 3). |
|
Recent values to use |
How many recent responses from the Toloker to use. If this field is not filled in, the calculation includes only task responses from the pool to which the rule is applied. If the field is filled in, the corresponding number of responses is used. The rule takes into account responses from both the current pool and other pools where this field is filled in. Learn more about how this field works. |
|
If |
A condition for performing the action in the then field:
To add multiple conditions, click . |
|
then |
Action to perform for the condition:
Learn more about how this field works. |
|
If |
A condition for performing the action in the then field:
To add multiple conditions, click . |
|
then |
Action to perform for the condition:
|
Examples of rules
Examples are provided for simple classification. There are 10 tasks per suite.
Warning
The assignments submitted by banned Tolokers will be taken into account if they are not rejected using manual review. They can be reassigned by setting up the Recompletion of assignments from banned users rule.
Calculating a skill and banning for incorrect responses
Both rules work independently:
- If the Toloker gives at least 3 responses to the tasks, the percentage of correct answers is written as the skill value.
- If the Toloker gives at least 3 answers to tasks and the percentage of correct answers is less than 65%, they are banned on the project.
The calculation uses up to 10 Toloker's responses to the project tasks.
The Toloker is blocked after the first incorrect response to the first, second or third task. However, the skill isn't set. Since the ban reason is not specified, there is no way to find out why the Toloker is banned.
All rules are applied independently:
-
If the Toloker gives at least 3 responses to the tasks, the percentage of correct answers is written as the skill value.
-
If the Toloker gives 2 incorrect responses to 3 tasks, they are blocked in the pool for 10 days.
-
If the Toloker gives 2 incorrect responses to 4 tasks, they are blocked in the pool for 10 days.
-
If the Toloker gives 5 or more responses to tasks and the percentage of correct responses is less than 80%, they are blocked in the pool for 10 days.
A set of rules like this prevents Tolokers from being banned for one incorrect response and lets you maintain high accuracy.
Setting the skill
If the Toloker completes 3 tasks, the skill is set to the percentage of correct answers.
Use the skill value to set access to other pools with filters.
This rule will never take effect because the number of responses counted (Recent values to use) is less than the number of responses in the rule (number of responses).
Blocking for incorrect responses
If the percentage of correct responses is less than 40%, the Toloker is blocked on the project for 30 days.
If the percentage of correct responses is less than 40%, the Toloker is blocked on the project for 30 days. The rule will be applied once, after the fifth response.
See also
For developers
Troubleshooting
It is better to use one skill in a project. You can choose the way to calculate the skill:
-
Calculate the skill for each pool separately. The current skill value is the value of the skill in the pool the Toloker completed last. This option is convenient if:
-
The pools are intended for different groups of Tolokers (for example, there are filters by city or country).
-
Pools are started one by one and you don't want to take into account the responses in the previous pools to calculate the skill in the current pool.
This calculation method is used by default when adding a quality control rule to a pool. For the control tasks block, leave the Recent control task responses to use field empty.
-
-
Calculate skill based on all tasks in a project This option is good if the pools are small and you don't need to have skill calculated for each pool.
This option is available only for skills on control tasks. To use it, fill in the Recent control task responses to use field in pool quality control rules.
Yes, of course — you can use the same skill for different projects. But most often, a skill is intended for a specific project. If the Toloker completes a certain task well, this doesn't mean that they will complete other ones successfully. Another disadvantage is that if you filter by skills that were set long ago, you will artificially limit the number of available Tolokers.
To perform actions with users (assign a skill or ban them) based on the majority vote, add a relevant rule to the pool.
Don't forget to enable Keep task order in the pool parameters. Majority vote is used in the projects with preset options (radio buttons or checkboxes). This rule won't apply to the text entry or file upload fields.
Yes, unfortunately, this can happen. This is why we recommend that you offer a training task or exam before the general task. In this case, only those people who showed good performance at the previous stage are selected for the main pool.
All responses to the task are taken into account. If one response differs from the majority vote, the whole task is counted as mismatching the responses of other Tolokers.