A category of components used in Template Builder that perform actions, such as open a link, display a message, or play a video. Some components work with data. For example, they can change the value from
false. List of actions
Tolokers with access to the pool who took tasks during the last hour.
A pool that was moved to the archive. Archived pools can't be started or edited and are not available for the assignment review. The pool is automatically archived if there is no activity in it for a month. With the archived pool, you can view the pool statistics, download the pool data, and clone the pool.
A task suite that a Toloker decides to pick up.
Blocking a Toloker's access to one or more projects. This lets you control which Tolokers will complete tasks. For example, you can choose all Tolokers with a skill value lower than
N and block their access to tasks. You can also unblock access.
Datasets that you send for labeling at a time. Batches are used in bespoke solutions.
See Control task.
Extra payment that you can give to Tolokers for completing tasks well. It improves motivation and encourages Tolokers to approach your tasks with care. You can give bonuses to one or more Tolokers (in addition to payment for completed tasks). The bonus amount can be from $0.01 to $100 per Toloker per time.
A type of challenge–response test to determine whether the user is human. Protects your project against fake users (robots). Restricts access to tasks for Tolokers who fail the test several times in a row.
Companies that take on the launch and implementation of your project and help you process the results. These companies have already integrated crowdsourcing into their business processes and offer excellent solutions.
JSON objects with a specified structure which help you build a task interface in Template Builder. Components are categorized depending on their purpose, such as data entry fields, conditions, or actions. The name of the component is specified in the
type property in the
<category>.<name> format. List of components
A description of your interface in the JSON format. You write the code for it in the Template Builder editor panel called Config.
A task that contains the correct response. Control tasks are used to monitor the quality of the Toloker's responses.
For example, you can track the percentage of correct responses and block Tolokers who respond poorly. To do this, use the Control tasks quality control rule.
The practice of obtaining information or services from a large dispersed group of people. These people perform micro tasks, and their collective work provides valuable input. For example, they can classify images into categories, and the resulting data can be used later to improve search relevance or recommendation systems.
Comma-separated values. A text file format in which data is separated by the comma, and lines are separated by a newline symbol.
A process of dividing a large project into several small ones. Decomposition results in shorter tasks that are easier to do, which helps improve quality.
A type of overlap that changes the number of Tolokers to complete each item in the pool. This number depends on confidence in the results: how well the Tolokers cope with the tasks or how consistent their responses are. The dynamic overlap value increases with the decreasing confidence in the results, and vice versa.
A quality control rule that restricts access to tasks for Tolokers who respond too quickly. You can use it to:
A percentage of the cost of tasks for which the requester is charged by the service when using Toloka.
A task that is performed at a physical location using the mobile app. It usually contains a point on the map that the Toloker needs to go to, check something, and take photos.
Criteria for choosing Tolokers. For example, you can select only Tolokers who speak English or have a certain skill.
A task without hints or the correct response. These are the general tasks you upload to the pool for which the Tolokers provide their responses.
See Control task.
See Control task.
The last task suite which contains a number of general tasks which is insufficient to form a complete suite. In this case, Tolokers will see a page with fewer general tasks and the full number of control and training tasks. Incomplete task suites can be enabled for tasks uploaded using smart mixing.
See Dynamic overlap.
The format of input and output data that you set in the Data specification field of the Task interface section when setting up a project.
data.inputcomponent of Template Builder to access the input data.
data.outputcomponent of Template Builder to access the output data.
See the Template Builder Read and write instructions to learn about working with data.
Project instructions that Tolokers see when choosing and completing the task. The clarity and completeness of the instructions affect response quality and the project rating. Good instructions help Tolokers complete the task correctly.
The number of Tolokers who started at least one task suite.
The data available only from within the task. This data is not saved to the results. Use it to calculate or store intermediate values. To access the internal data, use the
data.internal component of Template Builder.
See Dynamic overlap.
Corresponds to the final labeled data unit or a unique data unit to be labelled in Toloka. It is calculated as the aggregated result of all responses received for a task.
Adding informative tags to the data. For example, indicate if there are certain objects in a photo (streetlights, cars, pedestrians), transcribe an audio file, categorize a text or a video file, and add other similar information.
A set of languages in the Translations section into which the project interface can be translated.
A quality control rule that considers a response correct if it is chosen by the majority of Tolokers (also known as consensus). Other responses are considered incorrect.
The conversion of images of typed, handwritten or printed text into machine-encoded text, whether from a scanned document, a photo of a document, a scene-photo or from subtitle text superimposed on an image.
A set of data to label in a project. You can use the pool settings to select Tolokers, set the price for tasks, and add various quality control rules. Tasks in a pool share the same properties (such as Toloker selection settings and payment per task suite) and are sent out for completion at the same time.
A specific data labeling goal. This could be to moderate comments, to classify images, to transcribe audio recordings, or something else. A project consists of pools with tasks. Project settings define the task interface, instructions, and the input and output data shared by all pools within the same project.
A monitoring and control system that lets you get more accurate responses and restrict access to tasks for bad actors. Quality control consists of rules that are independent from each other. Learn more
Predefined quality control rule combinations available in the Quality control section of the pool settings. Three main quality control presets are available: elementary, basic, and advanced.
Independent rules that make up quality control. In the pool settings, the Quality control section allows you to set rules and configure them.
A quality control rule that can be configured to resend tasks for recompletion to other Tolokers if:
All pool tasks completed by this Toloker will be resent to other Tolokers. Tasks completed by banned Tolokers are reassigned, but those completed by paused Tolokers are not.
A pool that helps Tolokers who make mistakes improve their skills and get a second chance to complete tasks.
Number of days for checking Toloker responses manually.
The Toloka testing environment. This is where you can test your project settings as a Toloker before moving them to the Toloka production version and running the project for real Tolokers. This helps you avoid making mistakes and spending money on a task that isn't working right.
Side-by-side comparison type of projects. Toloka has several presets to compare objects side-by-side: images, video and audio files.
An assessment of the Toloker's quality of task completion on a scale of 0 to 100. The skill can be calculated automatically (for example, based on correct responses to control tasks), or assigned manually. Skills are used to select Tolokers in a pool. Learn more
A way to randomly combine tasks and generate task suites so that tasks aren't repeated and look different for each new Toloker.
A pool statistics parameters which shows the number of Tolokers who completed and submitted at least one task suite in the pool.
A request or question that elicits a response from an annotator in Toloka. A task is created for a single data unit that requires labeling, like a photo to classify, or a comment to review. It is a part of a web page (a task suite) that can contain various objects: images, text, input fields, and others. Tolokers read the instructions for the task, complete it and enter responses. In Template Builder, you can create an interface for such tasks.
The appearance of a task for Tolokers and the logic for processing responses.
Adding correct responses and hints to the tasks and changing task types. You can transform a general task into a control task by adding the correct task response, or into a training task by adding the correct response and a hint. Task markup is available only for training pools and pools uploaded with “smart mixing”. Learn more
The result of completing a single task by a Toloker. The task responses can be either correct or incorrect. You can block Tolokers who give too many incorrect task responses.
A group of several tasks given to a Toloker to submit all at once. For example, you can show four images on the same page. If the tasks are simple, you can add 10–20 tasks per suite. Don't make task suites too long because it slows down loading speed for Tolokers. A task suite may consist of a single task.
The result of completing a task suite by a Toloker. The task suite responses can be either correct or incorrect. You can decline incorrect responses using the assignment review and even block Tolokers who give too many incorrect responses.
The application programming interface which describes methods (a set of classes, procedures, functions, structures, or constants) used by Toloka to interact with other computer programs, and allows you to create tasks and receive user responses. Learn more
Any person from around the world who completes tasks in Toloka and gets paid for that. Collectively, they are usually referred to as "the crowd".
A pool of zero-price tasks for training Tolokers. Training tasks contain correct responses and hints that are shown if the Toloker answers incorrectly.
To improve the quality of results, you can add a training pool and choose only those Tolokers who pass training successfully. Learn more
A skill that is automatically created when the first training pool is added to the project. The skill name is
<project name> - training. The value is the percentage of correct task responses in the training pool. The skill is assigned to the Toloker after they complete the training pool.
There is only one training skill for the entire project. If you create new training pools, they will use the same skill. Learn more
A task that contains the correct response and a hint. If the Toloker responds incorrectly, they will see a hint and won't be able to go to the next task suite until they complete all tasks correctly. Only the first response to the task is taken into account for quality control and skill calculation.
Training tasks can be used in zero-price training pools to choose Tolokers who pass training.
They can also be used in regular pools as control tasks with hints. For example, you can track the percentage of correct task responses and ban Tolokers who respond poorly. To do this, use the Control tasks quality control rule.
Tab-separated values. A text file format in which data is separated by a tab, and lines are separated by a newline symbol.
A spreadsheet file format developed by Microsoft to replace the outdated XLS file type.
Last updated: March 24, 2023