- Dynamic overlap (incremental relabeling, IRL)
A type of overlap that changes the number of performers assigned to each pool task. This number depends on how well the performers cope with the task or how consistent their responses are.
A part of a web page that can contain various objects: images, text, input fields, and others. The Toloka user reads the instructions for the task, completes it and enters responses.
Settings for creating tasks:
Parameters for objects that are shown in the task suite (images, text, and so on);
Parameters for input fields.
The visual appearance of tasks.
Task suites that share the same properties (such as user selection settings and payment per task suite) and are sent out for completion at the same time.
The number of users performing the same task (overlap is used to make sure the results are reliable).
- Task suite
- Tasks placed on the same page. A task suite may consist of a single task.
- Training tasks
Tasks for training users.
Training tasks contain correct responses and hints. If a user responds incorrectly, a hint is displayed on the screen. The user must enter the correct response before continuing to the next task.
- Control tasks
Tasks with correct responses. They are used for controlling the quality of responses.
- Majority vote
A quality control approach that considers a response correct if it is chosen by the majority of users (also known as consensus). Other responses are considered incorrect.
An assessment of the user's performance (a number from 0 to 100) that can change as tasks are completed. Skills are used for user selection in a pool.
- Checking completed tasks
Manually checking the responses received by the requester. The requester accepts or rejects the responses. Rejected responses aren't paid for.