Glossary

A

Action components (component type for Template Builder)

A category of components used in Template Builder that perform actions, such as open a link, display a message, or play a video. Some components work with data. For example, they can change the value from true to false. List of actions

Active Tolokers

Tolokers with access to the pool who took tasks during the last hour.

Aggregated results

See Item.

Archived pool

A pool that was moved to the archive. Archived pools can't be started or edited and are not available for the assignment review. The pool is automatically archived if there is no activity in it for a month. With the archived pool, you can view the pool statistics, download the pool data, and clone the pool.

Assignment

A task suite that a Toloker decides to pick up.

Assignment statuses

B

Banning Tolokers

Blocking a Toloker's access to one or more projects. This lets you control which Tolokers will complete tasks. For example, you can choose all Tolokers with a skill value lower than N and block their access to tasks. You can also unblock access.

Batch (in bespoke solutions)

Datasets that you send for labeling at a time. Batches are used in bespoke solutions.

Benchmark task

See Control task.

Bonus

Extra payment that you can give to Tolokers for completing tasks well. It improves motivation and encourages Tolokers to approach your tasks with care. You can give bonuses to one or more Tolokers (in addition to payment for completed tasks). The bonus amount can be from $0.01 to $100 per Toloker per time.

C

Captcha

A type of challenge–response test to determine whether the user is human. Protects your project against fake users (robots). Restricts access to tasks for Tolokers who fail the test several times in a row.

Certified partners

Companies that take on the launch and implementation of your project and help you process the results. These companies have already integrated crowdsourcing into their business processes and offer excellent solutions.

Component (component type for Template Builder)

JSON objects with a specified structure which help you build a task interface in Template Builder. Components are categorized depending on their purpose, such as data entry fields, conditions, or actions. The name of the component is specified in the type property in the <category>.<name> format. List of components

Condition components (component type for Template Builder)

A category of components used in Template Builder that check whether the expression matches the specified condition. You can use them to check that the text is entered in a field. List of conditions

Configuration

A description of your interface in the JSON format. You write the code for it in the Template Builder editor panel called Config.

Control task (gold question, honeypot, benchmark task)

A task that contains the correct response. Control tasks are used to monitor the quality of the Toloker's responses.

For example, you can track the percentage of correct responses and block Tolokers who respond poorly. To do this, use the Control tasks quality control rule.

In addition to control tasks, there are general and training tasks.

Crowdsourcing

The practice of obtaining information or services from a large dispersed group of people. These people perform micro tasks, and their collective work provides valuable input. For example, they can classify images into categories, and the resulting data can be used later to improve search relevance or recommendation systems.

Crowdsourcing experts

Professionals who help create and configure projects, post tasks, and much more. They have passed a certification exam to prove their knowledge of crowdsourcing and earned a registered e-certificate.

CSV

Comma-separated values. A text file format in which data is separated by the comma, and lines are separated by a newline symbol.

D

Data entry field components (component type for Template Builder)

A category of components used in Template Builder that create data entry fields, such as text fields or drop-down lists. List of data entry fields

Decomposition

A process of dividing a large project into several small ones. Decomposition results in shorter tasks that are easier to do, which helps improve quality.

Dynamic overlap (incremental relabeling, IRL)

A type of overlap that changes the number of Tolokers to complete each item in the pool. This number depends on confidence in the results: how well the Tolokers cope with the tasks or how consistent their responses are. The dynamic overlap value increases with the decreasing confidence in the results, and vice versa.

Dynamic pricing

An option to set prices that depend on the Toloker's skill. This allows you, for example, to pay more to those who have a higher skill. Learn more

E

Earnings (quality control rule name)

A quality control rule that restricts the Toloker's earnings in the pool for a day. This allows you to get responses from as many Tolokers as possible and provide extra protection against robots.

Element layout components (component type for Template Builder)

A category of components used in Template Builder that arrange the interface elements, such as in columns or side-by-side. List of layout options

Exam

A pool with control tasks for choosing Tolokers. You can assign skills to Tolokers depending on whether their responses are correct and identify the best Tolokers. Learn more about exams.

F

Fast responses (quality control rule name)

A quality control rule that restricts access to tasks for Tolokers who respond too quickly. You can use it to:

  • Suspend access for Tolokers who respond randomly. You can set a reasonable minimal time required to complete a task suite.
  • Provide protection from robots (in this case, the time for completing the task suite should be much less).
Fee

A percentage of the cost of tasks for which the requester is charged by the service when using Toloka.

Field task

A task that is performed at a physical location using the mobile app. It usually contains a point on the map that the Toloker needs to go to, check something, and take photos.

File with tasks

A file that contains the data you need to label. Requesters use such files to upload tasks. Toloka allows using TSV, XLSX, or JSON files.

Filters

Criteria for choosing Tolokers. For example, you can select only Tolokers who speak English or have a certain skill.

You can use filters to choose Tolokers when setting up pools (in the Audience section), on the Users page, and for sending messages to a group of users. Learn more

Final answer

See Item.

Final label

See Item.

G

General task

A task without hints or the correct response. These are the general tasks you upload to the pool for which the Tolokers provide their responses.

In addition, there are control tasks (which contain the correct responses) and training tasks (which contain the correct responses and hints).

Gold question

See Control task.

H

Helper components (component type for Template Builder)

A category of components used in Template Builder for auxiliary operations, such as working with arrays. List of helpers

Honeypot

See Control task.

I

Incomplete task suite

The last task suite which contains a number of general tasks which is insufficient to form a complete suite. In this case, Tolokers will see a page with fewer general tasks and the full number of control and training tasks. Incomplete task suites can be enabled for tasks uploaded using smart mixing.

Incremental relabeling

See Dynamic overlap.

Input and output data

The format of input and output data that you set in the Data specification field of the Task interface section when setting up a project.

  • Input data is the source data you want to display or use. For example, links to images that will be shown to Tolokers. Use the data.input component of Template Builder to access the input data.
  • Output data is the data you receive after the task is completed, like the Tolokers' responses to your questions. Use the data.output component of Template Builder to access the output data.

See the Template Builder Read and write instructions to learn about working with data.

Instructions

Project instructions that Tolokers see when choosing and completing the task. The clarity and completeness of the instructions affect response quality and the project rating. Good instructions help Tolokers complete the task correctly.

Interested Tolokers

The number of Tolokers who started at least one task suite.

Internal data (used with Template Builder)

The data available only from within the task. This data is not saved to the results. Use it to calculate or store intermediate values. To access the internal data, use the data.internal component of Template Builder.

IRL (incremental labeling)

See Dynamic overlap.

Item (final answer, aggregated results, final label)

Corresponds to the final labeled data unit or a unique data unit to be labelled in Toloka. It is calculated as the aggregated result of all responses received for a task.

J

JSON

JavaScript Object Notation. A text file format used to store and transfer structured data.

K

Keep task order from uploaded data

Distributing the task suites in the pool in the same order as they are in the uploaded file. The tasks within each suite are shuffled.

L

Labeling

Adding informative tags to the data. For example, indicate if there are certain objects in a photo (streetlights, cars, pedestrians), transcribe an audio file, categorize a text or a video file, and add other similar information.

List of target languages

A set of languages in the Translations section into which the project interface can be translated.

M

Majority vote (quality control rule)

A quality control rule that considers a response correct if it is chosen by the majority of Tolokers (also known as consensus). Other responses are considered incorrect.

O

OCR

See Optical character recognition.

Optical character recognition (OCR)

The conversion of images of typed, handwritten or printed text into machine-encoded text, whether from a scanned document, a photo of a document, a scene-photo or from subtitle text superimposed on an image.

Overlap

The number of Tolokers who should complete each item in the pool. Overlap is used to make sure the results are reliable.

P

Plugin components (component type for Template Builder)

A category of components used in Template Builder that enable advanced features. For example, plugin.hotkeys lets you set up shortcuts. List of plugins

Pool

A set of data to label in a project. You can use the pool settings to select Tolokers, set the price for tasks, and add various quality control rules. Tasks in a pool share the same properties (such as Toloker selection settings and payment per task suite) and are sent out for completion at the same time.

Project

A specific data labeling goal. This could be to moderate comments, to classify images, to transcribe audio recordings, or something else. A project consists of pools with tasks. Project settings define the task interface, instructions, and the input and output data shared by all pools within the same project.

Project preset

A pre-configured project that contains description, instruction, predefined input and output data fields, and the task interface which you can edit.

Q

Quality control

A monitoring and control system that lets you get more accurate responses and restrict access to tasks for bad actors. Quality control consists of rules that are independent from each other. Learn more

Quality control presets

Predefined quality control rule combinations available in the Quality control section of the pool settings. Three main quality control presets are available: elementary, basic, and advanced.

Quality control rule

Independent rules that make up quality control. In the pool settings, the Quality control section allows you to set rules and configure them.

R

Recompletion of assignments from banned users (quality control rule name)

A quality control rule that can be configured to resend tasks for recompletion to other Tolokers if:

All pool tasks completed by this Toloker will be resent to other Tolokers. Tasks completed by banned Tolokers are reassigned, but those completed by paused Tolokers are not.

Retry pool

A pool that helps Tolokers who make mistakes improve their skills and get a second chance to complete tasks.

Review period

Number of days for checking Toloker responses manually.

Review task responses manually

A pool setting that allows you to manually check responses from Tolokers. You can either accept correct responses or reject poorly completed tasks and not pay for them.

S

Sandbox

The Toloka testing environment. This is where you can test your project settings as a Toloker before moving them to the Toloka production version and running the project for real Tolokers. This helps you avoid making mistakes and spending money on a task that isn't working right.

SbS

See Side-by-side.

Selective majority vote check

An option that allows you to selectively increase overlap for some items. This helps you save money and speed up pool completion. Learn more

Side-by-side (SbS)

Side-by-side comparison type of projects. Toloka has several presets to compare objects side-by-side: images, video and audio files.

Skill

An assessment of the Toloker's quality of task completion on a scale of 0 to 100. The skill can be calculated automatically (for example, based on correct responses to control tasks), or assigned manually. Skills are used to select Tolokers in a pool. Learn more

Skipped assignments (quality control rule name)

A quality control rule that restricts access to the pool tasks for Tolokers who skip several task suites in a row.

Smart mixing

A way to randomly combine tasks and generate task suites so that tasks aren't repeated and look different for each new Toloker.

Submitted in pool

A pool statistics parameters which shows the number of Tolokers who completed and submitted at least one task suite in the pool.

Submitted responses (quality control rule name)

A quality control rule that limits the number of task suites a Toloker can complete per day (taking overlap into account). This allows getting a broader selection of users.

T

Task

A request or question that elicits a response from an annotator in Toloka. A task is created for a single data unit that requires labeling, like a photo to classify, or a comment to review. It is a part of a web page (a task suite) that can contain various objects: images, text, input fields, and others. Tolokers read the instructions for the task, complete it and enter responses. In Template Builder, you can create an interface for such tasks.

Task interface

The appearance of a task for Tolokers and the logic for processing responses.

Task markup

Adding correct responses and hints to the tasks and changing task types. You can transform a general task into a control task by adding the correct task response, or into a training task by adding the correct response and a hint. Task markup is available only for training pools and pools uploaded with “smart mixing”. Learn more

Task response

The result of completing a single task by a Toloker. The task responses can be either correct or incorrect. You can block Tolokers who give too many incorrect task responses.

Task suite

A group of several tasks given to a Toloker to submit all at once. For example, you can show four images on the same page. If the tasks are simple, you can add 10–20 tasks per suite. Don't make task suites too long because it slows down loading speed for Tolokers. A task suite may consist of a single task.

Task suite response

The result of completing a task suite by a Toloker. The task suite responses can be either correct or incorrect. You can decline incorrect responses using the assignment review and even block Tolokers who give too many incorrect responses.

Toloka API

The application programming interface which describes methods (a set of classes, procedures, functions, structures, or constants) used by Toloka to interact with other computer programs, and allows you to create tasks and receive user responses. Learn more

The requests to the Toloka API are made using the HTTPS protocol. The service returns data in the JSON format.

Toloker

Any person from around the world who completes tasks in Toloka and gets paid for that. Collectively, they are usually referred to as "the crowd".

Training

A main pool that consists only of training tasks and can have zero cost. Use it to train Tolokers before performing real tasks if you can't use a training pool for your project.

Training pool

A pool of zero-price tasks for training Tolokers. Training tasks contain correct responses and hints that are shown if the Toloker answers incorrectly.

To improve the quality of results, you can add a training pool and choose only those Tolokers who pass training successfully. Learn more

Training skill

A skill that is automatically created when the first training pool is added to the project. The skill name is <project name> - training. The value is the percentage of correct task responses in the training pool. The skill is assigned to the Toloker after they complete the training pool.

There is only one training skill for the entire project. If you create new training pools, they will use the same skill. Learn more

Training task

A task that contains the correct response and a hint. If the Toloker responds incorrectly, they will see a hint and won't be able to go to the next task suite until they complete all tasks correctly. Only the first response to the task is taken into account for quality control and skill calculation.

Training tasks can be used in zero-price training pools to choose Tolokers who pass training.

They can also be used in regular pools as control tasks with hints. For example, you can track the percentage of correct task responses and ban Tolokers who respond poorly. To do this, use the Control tasks quality control rule.

TSV

Tab-separated values. A text file format in which data is separated by a tab, and lines are separated by a newline symbol.

V

View components (component type for Template Builder)

A category of components used in Template Builder that create visual interface elements. Examples include text, list, audio player, or image. List of views

X

XLSX

A spreadsheet file format developed by Microsoft to replace the outdated XLS file type.

Last updated: March 24, 2023

Introduction
Getting started
Important tips
Useful recommendations
Working with Toloka
Projects
Pools
Tasks
Results
Project analysis
Toloka settings
Task interface
Template Builder
HTML/CSS/JS editor
Help and support
FAQTroubleshootingSupportGlossary