Basic concepts

The terms are listed in alphabetical order.

Checking completed tasks

Manually checking the responses received by the requester. The requester accepts or rejects the responses. Rejected responses aren't paid for.

Control tasks

Tasks with correct responses. They are used for controlling the quality of responses.

Customizable project

Settings for creating tasks:

  • Parameters for objects that are shown in the task suite (images, text, and so on);

  • Parameters for input fields.

  • The visual appearance of tasks.

Use this project type if you have good knowledge of how Toloka works and need to configure your project as accurately as possible.

Dynamic overlap (incremental relabeling, IRL)

A type of overlap that changes the number of Tolokers assigned to each pool task. This number depends on how well the Tolokers cope with the task or how consistent their responses are.

Majority vote

A quality control approach that considers a response correct if it is chosen by the majority of Tolokers (also known as consensus). Other responses are considered incorrect.

Overlap

The number of Tolokers performing the same task (overlap is used to make sure the results are reliable).

Pool

Task suites that share the same properties (such as Toloker selection settings and payment per task suite) and are sent out for completion at the same time.

Skill

An assessment of the Toloker's performance (a number from 0 to 100) that can change as tasks are completed. Skills are used for Toloker selection in a pool.

Task

A part of a web page that can contain various objects: images, text, input fields, and others. The Toloka Toloker reads the instructions for the task, completes it and enters responses.

Task suite
Tasks placed on the same page. A task suite may consist of a single task.
Training tasks

Tasks for training Tolokers.

Training tasks contain correct responses and hints. If a Toloker responds incorrectly, a hint is displayed on the screen. The Toloker must enter the correct response before continuing to the next task.