Terms and definitions
- Toloka API
The Toloka API, Toloka application programming interface, describes methods (a set of classes, procedures, functions, structures, or constants) used by Toloka to interact with another computer program and allows you to create tasks and receive user responses (see the documentation).
- TSV file with tasks
TSV (tab-separated values) is a text file format in which data is separated by a tab, and lines are separated by a newline symbol.
- Active performers
- Archive pool
Pool, sent to the archive. Such pool can't be started, edite, and isn't available for the assignment review. The pool is automatically archived if none of the following actions is performed in the pool for a month:
- Blocking a performer
You can block a performer's access to one or more projects. This lets you control manually which performers will complete tasks. For example, you can choose all performers with a skill value lower than N and block their access to tasks. You can also unblock access.
You can give rewards to performers for completing tasks well. This improves motivation and encourages performers to approach your tasks with care. You can give rewards to one or more performers (in addition to payment for completed tasks). The reward amount can be from 0.01 to 100 dollars per performer per time.
- Fast responses
Restricting access to tasks for performers who respond too quickly. Used to:
- Suspend access for performers who cheat in their responses (in this case, set the time required to complete a task suite when giving random responses).
- Provide protection from robots (in this case, the time for completing the task suite should be 2 times less).
- Submitted in pool
The number of performers who completed at least one task suite.
- Input and output data
In the Specification field, you describe the format of input and output data.
Input data is the data you need for your tasks. For instance, this could be links to images that you want to show to performers.
Output is the data you receive after the task is completed, like the performer's answers to your questions.
- Selective majority vote check
Majority vote checks only some tasks. This allows you to save money and speeds up the pool completion.
- Completed tasks (quality control rule)
- Ready-made audience sets (filters)
Ready set of filters for choosing performers. For example, mobile users.
- Ready protection sets
Several quality control rules with default settings.
- Dynamic overlap
Dynamic overlap allows you to change the number of performers to complete each task in the pool depending on how well the performers cope with the task or how consistent their responses are.
- Dynamic pricing
- Task (Main task)
A task that doesn't provide hints or the correct response. The responses are provided by the performers. These are the main tasks you will upload to the pool to give to the performers.
- Interested users
The number of performers who started at least one task suite.
- Earnings (quality control rule)
Restriction of the performer's earnings in pool during the day, to get responses from as many performers as possible and provide protection from robots.
Task instructions the performer sees when choosing and completing the task. The clarity and completeness of the instructions affect the response quality and the project rating. Good instructions help the performer complete the task correctly.
- Task interface
The task interface defines the task appearance for the performer and the logic for processing responses.
- Captcha (quality control rule)
Restricting access to tasks for the performers who fail to enter captcha several times in a row. This helps protect against robots.
The requester is charged a fee for using Toloka — a percentage of the cost of tasks. It is 30%, but not less than $0.005.
- Control task
A task that contains the correct response. Control tasks are needed to monitor the quality of the performer's responses.
For example, you can track the percentage of correct responses and block performers who respond poorly. To do this, use the Control tasks rule.Note.
- Add at least 1% of control tasks to the pool.
- Design control tasks so that different versions of correct responses are used with the same frequency. Learn more.
- Quality control
Quality control lets you get more accurate responses and restrict access to tasks for cheating performers. Quality control consists of rules that are independent from each other. The quality control block allows you to set rules and configure them. Learn more.
- Majority vote (quality control rule)
A quality control method based on matching responses from the majority of performers who complete the same task.
Assessment of the performer's quality of task completion (a number from 0 to 100). The skill can be calculated automatically (for example, based on correct answers to control tasks), or assigned manually. Learn more.
- Incomplete page
The last page with an insufficient number of main tasks. It's generated if less than the minimum remaining number of main tasks are not completed. In this case, the number of control and training tasks must be complete.
- Training task
A task that contains the correct response and a hint. If the performer answers incorrectly, they will see a hint and won't be able to go to the next task suite until they complete all tasks correctly. Only the first response to the task is considered for quality control and skill calculation.
Training tasks can be used in free training pools to choose performers who passed training.
They can also be used in regular pools as control tasks with hints. For example, you can track the percentage of correct responses and block performers who respond poorly. To do this, use the Control tasks rule.
Training pool with free tasks for the performers to practice on. Training tasks contain correct responses and a hint shown if the performer answers incorrectly.
To improve the quality of results, you can add a training pool and choose only those performers who successfully completed the training. Learn more.
- Reviewing assignments
A pool setting that allows you to check responses and not to pay for poorly completed tasks.
The number of performers who should complete each task in the pool.
- Personalized quality forecast
A personalized quality forecast is based on a large amount of data about user behavior in the system, how other users completed your task, and the task itself.
Toloka testing environment. This is where you can test your project settings as a performer before moving them to the Toloka production version and running the project for real performers. This helps you avoid making mistakes and spending money on a task that isn't working right.
- Recompletion of assignments from banned users (quality control rule)
- A rule that can be configured to send tasks to other performers for re-execution if:
- Field task
A task to perform in the mobile app. It usually contains a point on the map for the performer to go to the location, check something, and take photos.
- Quality control rule
- Independent rules that make up the quality control. The quality control block allows you to set rules and configure them.
- Skipping tasks (quality control rule)
A set of paid tasks sent out for completion at the same time.
- Task markup
Task markup lets you add correct responses and hints to the tasks, as well as change task types. You can transform a main task in a control task by adding the correct response, or in a training task by specifying the correct response and a hint. Task markup is available only for training pools and pools uploaded with “smart mixing”.
- Retry pool
A "retry pool" will help performers who make mistakes improve their skills and get a se chance to complete tasks.
- Performer rating
The performer's rating reflects the quality of the performer's responses to tasks. It takes into account responses to control tasks, bans and periods without work. The rating is calculated as the weighted average value for the last few weeks. So the rating can “suddenly” decrease or increase.
- Certified partners
Companies that take on the launch and implementation of your project and help you process the results. These companies have already integrated crowdsourcing into their business processes and they offer excellent solutions.
- Keeping the task order
- Crowdsourcing experts
Professionals who help create and configure projects, post tasks and much more. They have passed a certification exam to prove their knowledge of crowdsourcing and earned a registered e-certificate from Yandex. To learn about terms of partnership, contact these experts directly.
- List of target languages
Used in the Translation to other languages section if there is no source language.Display order:
- Review period
Number of days to review the task.
- Task suite
A task suite can contain one or several tasks that are shown on the same page. If the tasks are simple, you can add 10-20 tasks per suite. Don't make task suites too long because it slows down loading speed for performers.
A main pool that consists only of training tasks and can have a zero cost.
Use it to train performers before performing real tasks if you can't use a training pool for your project.
- Training skill
A skill that is automatically created when the first training pool is added to the project. The skill name is
<project name> - training. The value is the percentage of correct responses in the training pool. The skill is assigned to the performer after they finish the training pool.
There is only one training skill for the entire project. If you create new training pools, they will use the same skill.
- Smart mixing
Smart mixing randomly generates pages with tasks so that tasks aren't repeated for each performer.
Criteria for choosing performers. For example, to choose only Russian-speaking performers or performers with some skill.
- Project template