Terms and definitions

Active performers

Performers who viewed or completed tasks during the last hour and match the filters specified in the pool settings. This is the number of performers who will see your task if you publish it now.

Archive pool

A pool that was moved to the archive. Archive pools can't be started or edited and are not available for assignment review. The pool is automatically archived if there is no activity in it for a month.

Banning performers

You can block a performer's access to one or more projects. This lets you control manually which performers will complete tasks. For example, you can choose all performers with a skill value lower than N and block their access to tasks. You can also unblock access.

Captcha (quality control rule)

Restricts access to tasks for performers who fail to enter captcha several times in a row. Protects your project against fake users (robots).

Certified partners

Companies that take on the launch and implementation of your project and help you process the results. These companies have already integrated crowdsourcing into their business processes and offer excellent solutions.

Completed tasks (quality control rule)

Number of completed task suites (taking overlap into account).

Control task

A task that contains the correct response. Control tasks are needed to monitor the quality of the performer's responses.

For example, you can track the percentage of correct responses and block performers who respond poorly. To do this, use the Control tasks rule.

Note.

In addition to control tasks, there are main and training tasks.

Crowdsourcing experts

Professionals who help create and configure customizable projects, post tasks, and much more. They have passed a certification exam to prove their knowledge of crowdsourcing and earned a registered e-certificate from Yandex. To learn about the terms of partnership, contact these experts directly.

Customizable project

A customizable project consists of pools with tasks. Project settings define the task interface and the input and output data.

Dynamic overlap

Dynamic overlap allows you to change the number of performers to complete each task in the pool depending on how well the performers cope with the task or how consistent their responses are.

Dynamic pricing

Allows you to additionally set other prices that depend on the performer's skill. For example, to pay more to those who have a higher skill. Learn more.

Earnings (quality control rule)

Restricts the performer's earnings in the pool for a day to get responses from as many performers as possible and provide protection against robots.

Exam

A pool with control tasks for choosing performers. You can assign skills to performers depending on whether their responses are correct and identify the best performers.

Fast responses

Restricts access to tasks for performers who respond too quickly. You can use it to:

  • Suspend access for performers who cheat in their responses (in this case, set the time required to complete a task suite when giving random responses).
  • Provide protection from robots (in this case, the time for completing the task suite should be 2 times less).
Fee

The requester is charged a fee for using Toloka — a percentage of the cost of tasks. It is 30%, but not less than $0.005.

Field task

A task to be performed in the mobile app. It usually contains a point on the map that the performer needs to go to, check something, and take photos.

Filters

Criteria for choosing performers. For example, you can select only performers who speak English or have a certain skill.

Filters are used in the pool settings on the Users page and for sending messages to a group of users. Learn more.

Incomplete task suite

The last suite with an insufficient number of main tasks. This is generated if less than the minimum remaining number of general tasks are not completed. In this case, the number of control and training tasks must be complete.

Input and output data

The format of input and output data is set in the Specifications field.

Input data is the data you need for your tasks. For example, links to images that you want to show to performers.

Output data is the data you receive after the task is completed. For example, the performers' responses to your questions.

Instructions

Task instructions that the performer sees when choosing and completing the task. The clarity and completeness of the instructions affect response quality and the customizable project's rating. Good instructions help the performer complete the task correctly.

Interested users

The number of performers who started at least one task suite.

Keeping the task order

Task suites in the pool are distributed in the same order as they are in the uploaded TSV file. The tasks within each suite are shuffled. Learn more.

List of target languages

Used in the Translations section if there is no source language.

Display order:
  • English
  • Russian
  • Turkish
  • French
  • Indonesian
  • Vietnamese
  • Uzbek
  • Spanish
  • German
  • Italian
  • Polish
Majority vote (quality control rule)

A quality control method based on matching responses from the majority of performers who complete the same task.

Overlap

The number of performers who should complete each task in the pool.

Performer rating

The performer's rating reflects the quality of the performer's responses to tasks. It takes into account responses to control tasks, bans, and periods without work. The rating is calculated as the weighted average value for the last few weeks. So the rating can “suddenly” decrease or increase.

Personalized quality forecast

A personalized quality forecast is based on a large amount of data about user behavior in the system, how other users completed your task, and the task itself.

Pool

A set of paid tasks sent out for completion at the same time.

Training

A main pool that consists only of training tasks and can have zero cost.

Use it to train performers before performing real tasks if you can't use a training pool for your project.

Customizable project template

The template contains pre-configured input and output data fields and the task interface, which you can edit.

Quality control

Quality control lets you get more accurate responses and restrict access to tasks for cheating performers. Quality control consists of rules that are independent from each other. The quality control block allows you to set rules and configure them. Learn more.

Quality control rule
Independent rules that make up quality control. The quality control block allows you to set rules and configure them.
Ready protection sets

Several quality control rules with default settings.

Recompletion of assignments from banned users (quality control rule)
A rule that can be configured to resend tasks for recompletion to other performers if:All pool tasks completed by this performer will be resent to other performers. Tasks completed by banned performers are reassigned, but those completed by paused performers are not.
Retry pool

The retry pool helps performers who make mistakes improve their skills and get a second chance to complete tasks.

Review period

Number of days for checking the task.

Assignment review

A pool setting that allows you to check responses so you don't have to pay for poorly completed tasks.

Reward

You can give rewards to performers for completing tasks well. This improves motivation and encourages performers to approach your tasks with care. You can give rewards to one or more performers (in addition to payment for completed tasks). The reward amount can be from $0.01 to $100 per performer per time.

Sandbox

Toloka testing environment. This is where you can test your project settings as a performer before moving them to the Toloka production version and running the project for real performers. This helps you avoid making mistakes and spending money on a task that isn't working right.

Selective majority vote check

Majority vote checks only some tasks. This helps you save money and speed up pool completion.

Skill

Assessment of the performer's quality of task completion on a scale of 0 to 100. The skill can be calculated automatically (for example, based on correct responses to control tasks), or assigned manually. Learn more.

Skipping tasks (quality control rule)

Restricts access to the pool tasks for performers who skip several task suites in a row.

Smart mixing

Smart mixing randomly generates task suites so that tasks aren't repeated for each performer.

Submitted in pool

The number of performers who completed at least one task suite.

TSV file with tasks

TSV (tab-separated values) is a text file format in which data is separated by a tab, and lines are separated by a newline symbol.

Task (general task)

A task that doesn't provide hints or the correct response. The responses are provided by the performers. These are the general tasks you upload to the pool to give to the performers.

In addition, there are control and training tasks.

Task interface

The task interface defines the task appearance for the performer and the logic for processing responses.

Task markup

Task markup lets you add correct responses and hints to the tasks, as well as change task types. You can transform a general task into a control task by adding the correct response, or into a training task by adding the correct response and a hint. Task markup is available only for training pools and pools uploaded with smart mixing.

Learn more about task markup.

Task suite

A task suite contains one or several tasks that are shown on the same page. If the tasks are simple, you can add 10-20 tasks per suite. Don't make task suites too long because it slows down loading speed for performers.

Toloka API

The Toloka API (Toloka application programming interface) describes methods (a set of classes, procedures, functions, structures, or constants) used by Toloka to interact with other computer programs, and allows you to create tasks and receive user responses (see the documentation).

Training

A pool of zero-price tasks for training performers. Training tasks contain correct responses and hints that are shown if the performer answers incorrectly.

To improve the quality of results, you can add a training pool and choose only those performers who pass training successfully. Learn more.

Training skill

A skill that is automatically created when the first training pool is added to the project. The skill name is <project name> - training. The value is the percentage of correct responses in the training pool. The skill is assigned to the performer after they complete the training pool.

There is only one training skill for the entire project. If you create new training pools, they will use the same skill.

Training task

A task that contains the correct response and a hint. If the performer answers incorrectly, they will see a hint and won't be able to go to the next task suite until they complete all tasks correctly. Only the first response to the task is taken into account for quality control and skill&#x08; calculation.

Training tasks can be used in zero-price training pools to choose performers who pass training.

They can also be used in regular pools as control tasks with hints. For example, you can track the percentage of correct responses and ban performers who respond poorly. To do this, use the Control tasks rule.