A category of components used in Template Builder that perform actions, such as open a link, display a message, or play a video. Some components work with data. For example, they can change the value from true to false. List of actions
Tolokers with access to the pool who took tasks during the last hour.
See Item.
A pool that was moved to the archive. Archived pools can't be started or edited and are not available for the assignment review. The pool is automatically archived if there is no activity in it for a month. With the archived pool, you can view the pool statistics, download the pool data, and clone the pool.
A task suite that a Toloker decides to pick up.
Blocking a Toloker's access to one or more projects. This lets you control which Tolokers will complete tasks. For example, you can choose all Tolokers with a skill value lower than N and block their access to tasks. You can also unblock access.
Datasets that you send for labeling at a time. Batches are used in bespoke solutions.
See Control task.
Extra payment that you can give to Tolokers for completing tasks well. It improves motivation and encourages Tolokers to approach your tasks with care. You can give bonuses to one or more Tolokers (in addition to payment for completed tasks). The bonus amount can be from $0.01 to $100 per Toloker per time.
A type of challenge–response test to determine whether the user is human. Protects your project against fake users (robots). Restricts access to tasks for Tolokers who fail the test several times in a row.
Companies that take on the launch and implementation of your project and help you process the results. These companies have already integrated crowdsourcing into their business processes and offer excellent solutions.
JSON objects with a specified structure which help you build a task interface in Template Builder. Components are categorized depending on their purpose, such as data entry fields, conditions, or actions. The name of the component is specified in the type property in the <category>.<name> format. List of components
A category of components used in Template Builder that check whether the expression matches the specified condition. You can use them to check that the text is entered in a field. List of conditions
A description of your interface in the JSON format. You write the code for it in the Template Builder editor panel called Config.
A task that contains the correct response. Control tasks are used to monitor the quality of the Toloker's responses.
For example, you can track the percentage of correct responses and block Tolokers who respond poorly. To do this, use the Control tasks quality control rule.
In addition to control tasks, there are general and training tasks.
The practice of obtaining information or services from a large dispersed group of people. These people perform micro tasks, and their collective work provides valuable input. For example, they can classify images into categories, and the resulting data can be used later to improve search relevance or recommendation systems.
Professionals who help create and configure projects, post tasks, and much more. They have passed a certification exam to prove their knowledge of crowdsourcing and earned a registered e-certificate.
Comma-separated values. A text file format in which data is separated by the comma, and lines are separated by a newline symbol.
A category of components used in Template Builder that create data entry fields, such as text fields or drop-down lists. List of data entry fields
A process of dividing a large project into several small ones. Decomposition results in shorter tasks that are easier to do, which helps improve quality.
A type of overlap that changes the number of Tolokers to complete each item in the pool. This number depends on confidence in the results: how well the Tolokers cope with the tasks or how consistent their responses are. The dynamic overlap value increases with the decreasing confidence in the results, and vice versa.
An option to set prices that depend on the Toloker's skill. This allows you, for example, to pay more to those who have a higher skill. Learn more
A quality control rule that restricts the Toloker's earnings in the pool for a day. This allows you to get responses from as many Tolokers as possible and provide extra protection against robots.
A category of components used in Template Builder that arrange the interface elements, such as in columns or side-by-side. List of layout options
A pool with control tasks for choosing Tolokers. You can assign skills to Tolokers depending on whether their responses are correct and identify the best Tolokers. Learn more about exams.
A quality control rule that restricts access to tasks for Tolokers who respond too quickly. You can use it to:
A percentage of the cost of tasks for which the requester is charged by the service when using Toloka.
A task that is performed at a physical location using the mobile app. It usually contains a point on the map that the Toloker needs to go to, check something, and take photos.
A file that contains the data you need to label. Requesters use such files to upload tasks. Toloka allows using TSV, XLSX, or JSON files.
Criteria for choosing Tolokers. For example, you can select only Tolokers who speak English or have a certain skill.
You can use filters to choose Tolokers when setting up pools (in the Audience section), on the Users page, and for sending messages to a group of users. Learn more
See Item.
See Item.
A task without hints or the correct response. These are the general tasks you upload to the pool for which the Tolokers provide their responses.
In addition, there are control tasks (which contain the correct responses) and training tasks (which contain the correct responses and hints).
See Control task.
A category of components used in Template Builder for auxiliary operations, such as working with arrays. List of helpers
See Control task.
The last task suite which contains a number of general tasks which is insufficient to form a complete suite. In this case, Tolokers will see a page with fewer general tasks and the full number of control and training tasks. Incomplete task suites can be enabled for tasks uploaded using smart mixing.
See Dynamic overlap.
The format of input and output data that you set in the Data specification field of the Task interface section when setting up a project.
data.input component of Template Builder to access the input data.data.output component of Template Builder to access the output data.See the Template Builder Read and write instructions to learn about working with data.
Project instructions that Tolokers see when choosing and completing the task. The clarity and completeness of the instructions affect response quality and the project rating. Good instructions help Tolokers complete the task correctly.
The number of Tolokers who started at least one task suite.
The data available only from within the task. This data is not saved to the results. Use it to calculate or store intermediate values. To access the internal data, use the data.internal component of Template Builder.
See Dynamic overlap.
Corresponds to the final labeled data unit or a unique data unit to be labelled in Toloka. It is calculated as the aggregated result of all responses received for a task.
JavaScript Object Notation. A text file format used to store and transfer structured data.
Distributing the task suites in the pool in the same order as they are in the uploaded file. The tasks within each suite are shuffled.
Adding informative tags to the data. For example, indicate if there are certain objects in a photo (streetlights, cars, pedestrians), transcribe an audio file, categorize a text or a video file, and add other similar information.
A set of languages in the Translations section into which the project interface can be translated.
A quality control rule that considers a response correct if it is chosen by the majority of Tolokers (also known as consensus). Other responses are considered incorrect.
See Optical character recognition.
The conversion of images of typed, handwritten or printed text into machine-encoded text, whether from a scanned document, a photo of a document, a scene-photo or from subtitle text superimposed on an image.
The number of Tolokers who should complete each item in the pool. Overlap is used to make sure the results are reliable.
A category of components used in Template Builder that enable advanced features. For example, plugin.hotkeys lets you set up shortcuts. List of plugins
A set of data to label in a project. You can use the pool settings to select Tolokers, set the price for tasks, and add various quality control rules. Tasks in a pool share the same properties (such as Toloker selection settings and payment per task suite) and are sent out for completion at the same time.
A specific data labeling goal. This could be to moderate comments, to classify images, to transcribe audio recordings, or something else. A project consists of pools with tasks. Project settings define the task interface, instructions, and the input and output data shared by all pools within the same project.
A pre-configured project that contains description, instruction, predefined input and output data fields, and the task interface which you can edit.
A monitoring and control system that lets you get more accurate responses and restrict access to tasks for bad actors. Quality control consists of rules that are independent from each other. Learn more
Predefined quality control rule combinations available in the Quality control section of the pool settings. Three main quality control presets are available: elementary, basic, and advanced.
Independent rules that make up quality control. In the pool settings, the Quality control section allows you to set rules and configure them.
A quality control rule that can be configured to resend tasks for recompletion to other Tolokers if:
All pool tasks completed by this Toloker will be resent to other Tolokers. Tasks completed by banned Tolokers are reassigned, but those completed by paused Tolokers are not.
A pool that helps Tolokers who make mistakes improve their skills and get a second chance to complete tasks.
Number of days for checking Toloker responses manually.
A pool setting that allows you to manually check responses from Tolokers. You can either accept correct responses or reject poorly completed tasks and not pay for them.
Starting July 1, 2024 Sandbox environment is unavailable.
The Toloka testing environment. This is where you can test your project settings as a Toloker before moving them to the Toloka production version and running the project for real Tolokers. This helps you avoid making mistakes and spending money on a task that isn't working right.
See Side-by-side.
An option that allows you to selectively increase overlap for some items. This helps you save money and speed up pool completion. Learn more
Side-by-side comparison type of projects. Toloka has several presets to compare objects side-by-side: images, video and audio files.
An assessment of the Toloker's quality of task completion on a scale of 0 to 100. The skill can be calculated automatically (for example, based on correct responses to control tasks), or assigned manually. Skills are used to select Tolokers in a pool. Learn more
A quality control rule that restricts access to the pool tasks for Tolokers who skip several task suites in a row.
A way to randomly combine tasks and generate task suites so that tasks aren't repeated and look different for each new Toloker.
A pool statistics parameters which shows the number of Tolokers who completed and submitted at least one task suite in the pool.
A quality control rule that limits the number of task suites a Toloker can complete per day (taking overlap into account). This allows getting a broader selection of users.
A request or question that elicits a response from an annotator in Toloka. A task is created for a single data unit that requires labeling, like a photo to classify, or a comment to review. It is a part of a web page (a task suite) that can contain various objects: images, text, input fields, and others. Tolokers read the instructions for the task, complete it and enter responses. In Template Builder, you can create an interface for such tasks.
The appearance of a task for Tolokers and the logic for processing responses.
Adding correct responses and hints to the tasks and changing task types. You can transform a general task into a control task by adding the correct task response, or into a training task by adding the correct response and a hint. Task markup is available only for training pools and pools uploaded with “smart mixing”. Learn more
The result of completing a single task by a Toloker. The task responses can be either correct or incorrect. You can block Tolokers who give too many incorrect task responses.
A group of several tasks given to a Toloker to submit all at once. For example, you can show four images on the same page. If the tasks are simple, you can add 10–20 tasks per suite. Don't make task suites too long because it slows down loading speed for Tolokers. A task suite may consist of a single task.
The result of completing a task suite by a Toloker. The task suite responses can be either correct or incorrect. You can decline incorrect responses using the assignment review and even block Tolokers who give too many incorrect responses.
The application programming interface which describes methods (a set of classes, procedures, functions, structures, or constants) used by Toloka to interact with other computer programs, and allows you to create tasks and receive user responses. Learn more
The requests to the Toloka API are made using the HTTPS protocol. The service returns data in the JSON format.
Any person from around the world who completes tasks in Toloka and gets paid for that. Collectively, they are usually referred to as "the crowd".
A main pool that consists only of training tasks and can have zero cost. Use it to train Tolokers before performing real tasks if you can't use a training pool for your project.
A pool of zero-price tasks for training Tolokers. Training tasks contain correct responses and hints that are shown if the Toloker answers incorrectly.
To improve the quality of results, you can add a training pool and choose only those Tolokers who pass training successfully. Learn more
A skill that is automatically created when the first training pool is added to the project. The skill name is <project name> - training. The value is the percentage of correct task responses in the training pool. The skill is assigned to the Toloker after they complete the training pool.
There is only one training skill for the entire project. If you create new training pools, they will use the same skill. Learn more
A task that contains the correct response and a hint. If the Toloker responds incorrectly, they will see a hint and won't be able to go to the next task suite until they complete all tasks correctly. Only the first response to the task is taken into account for quality control and skill calculation.
Training tasks can be used in zero-price training pools to choose Tolokers who pass training.
They can also be used in regular pools as control tasks with hints. For example, you can track the percentage of correct task responses and ban Tolokers who respond poorly. To do this, use the Control tasks quality control rule.
Tab-separated values. A text file format in which data is separated by a tab, and lines are separated by a newline symbol.
A category of components used in Template Builder that create visual interface elements. Examples include text, list, audio player, or image. List of views
A spreadsheet file format developed by Microsoft to replace the outdated XLS file type.
Last updated: March 24, 2023