Editing a pool
You can edit a pool with the “Open” or “Closed” status. While editing, an open pool switches to the “CLOSED_FOR_UPDATE” status. For example, this way you can change the payout amount in a running pool.
The cost of a task is registered at the moment when a Toloker accepts it. If the cost changes during task completion, the Toloker will receive the amount that was specified before editing the cost.
To edit the pool parameters, click Edit at the top of the pool page or in the list of pools on the project page.
Will be displayed instead of the project description in the task list for Tolokers and linked training.
If necessary, you can add a private comment that will only be available to you.
Price per task suite, $
Payment per task suite in US dollars. For cents, use the dot (".") as a separator. If the Pool type is General tasks, the minimum price per task suite is $0.005. For other pool types, you can set the price to zero. To learn more about price management, go to Setting up pricing.
Tolokers' interest in the price
Calculated automatically. Shows how the price meets Tolokers' expectations.
Number of tasks per suite
Calculated automatically. Shows the number of tasks per suite recommended for this type of project.
The number of Tolokers who should complete each task in the pool.
Configure this parameter if you need several people to complete your task. Overlap is necessary in tasks for collecting photo datasets, recording audio, or conducting surveys. In other types of tasks, it can improve the quality of results.
To save money, you can set up dynamic overlap (also known as incremental relabeling or IRL).
Dynamic overlap will help you save money. Learn more.
Tasks that contain some content which can be marked as adult, including shocking or pornographic content. These tasks are only assigned to Tolokers who have agreed to complete these types of tasks.
If you aren't sure whether the tasks have this kind of content, select the My tasks may contain shocking or pornographic content option.
Filters and skills
By setting the filters, you choose Tolokers that meet your requirements to complete your task. There are filters by various criteria, and filters by skills that you can create yourself.
You can filter Tolokers by language, country, city, and so on. Learn more
A setting for choosing Tolokers for your tasks. You can set the percentage or number of top-rated Tolokers.
When there are many Tolokers, you can get results quickly, but the quality of the Tolokers' responses may be poor. Find a balance that meets your needs. Learn more.
You can choose how often captchas are shown to Tolokers:
“No” — Don't show captchas.
“Low” — Show a captcha after every 20 assignments.
“Medium”/ “High” — Show a captcha after every 10 assignments.
To show the captcha to Tolokers, set the quality control rule.
New built-in captcha
Toloka introduces a new built-in captcha service which will eventually replace the old, project-based version. It is activated automatically and protects your projects from bots and other fraud while also improving overall labeling quality.
Review task responses manually
Set the time period when you can check the tasks and reject them if the results are poor.
Review period in days
The number of days for reviewing and accepting tasks (max: 21). The Toloker will see the deadline for checking the tasks:
In the task information on the Toloka main page.
In the history of completed tasks.
Quality control rules
You can add other quality control rules by clicking Add a quality control rule. You can select ready-made sets of rules or set the rules you need manually. Learn more.
Restricting access for Tolokers who respond too quickly. Specify the minimum time required for completing a task suite (in seconds) and set the condition.
For example: if the number of responses is > 5 and the number of quick responses is > 3, then ban the Toloker on the project for 5 days. Learn more.
Evaluate Tolokers' responses by comparing them with the majority vote: reward Tolokers who give correct responses and block those who often make mistakes.
In the Accept as majority field, specify the number of matching responses that is considered the majority vote and set the condition. The rule takes effect when the number of responses for the task is equal to the overlap.
For example: if the number of responses is > 5 and the percentage of correct responses is < 60, then ban the Toloker on the project for 10 days.
This field is only available if you have training pools in the project.
Training pool that will be linked to the main one.
This box is only available if you filled in the Training box.
Percentage of correct responses in training tasks (from 5 to 100) required to be admitted to the pool tasks. The calculation is based on the first response the Toloker gave in each task.
The minimum required level that you can set is 5. Tolokers who complete training with apercentage below this level won't have access to tasks.
Dynamic pricing and overlap
You set the range, the service analyzes the responses, their consistency, the level of Tolokers' skills and, if necessary, increases the overlap within the range you specified. Learn more.
Number from 0 to 100. Allows you to rank a pool within a project. First, a Toloker is assigned tasks from a pool with higher priority.
Time per task suite, sec
The time allowed for completing a task suite, in seconds. Uncompleted tasks are redistributed to other Tolokers.
We recommend spending no more than 60 seconds per task (including the time for page loading and sending responses).
If you set the interval too long, the tasks will stay open for a long time. If it is too short, the tasks won't be completed and paid for.
Keep pool open after completion, sec
The interval during which the pool will remain open from the moment all tasks are completed. Minimum — 0 (1 if you use dynamic overlap), maximum — 259200 seconds (three days).
For example, you can use this parameter if you need an open pool where you regularly upload new tasks.
Pool closing date
The date the pool closes. The pool closes automatically on this date, even if the tasks aren't marked up.
Specify the pool type:
General tasks: The main pool that contains your regular tasks.
Exam: A pool that contains only control tasks.
Retry: A pool that helps Tolokers who make mistakes improve their skills and get a second chance to complete tasks.
Other: If none of the previous options fit.
If the price per task suite is zero, you must select the pool type.
Keep task order
Option disabled (default value)
The uploaded tasks are grouped in pages and given to Tolokers in random order. Within the page, the task order is always random.
If you specified 2 tasks per suite when uploading the file with image links to the pool, the system can generate them as follows:
|Tasks in the file||Page 1||Page 2||Page 3|
|Image 1||Image 2||Image 6||Image 4|
|Image 2||Image 5||Image 1||Image 3|
Tasks will be grouped on pages in the order they are listed in the task file.
If you specified 2 tasks per suite when uploading the file with image links to the pool, you're equally likely to get pages where the first link goes first and the second goes second, and the other way around:
|Tasks in the file||Page 1||Page 2||Page 3|
|Image 1||Image 1||Image 3||Image 5|
|Image 2||Image 2||Image 4||Image 6|
|Image 4||Image 2||Image 4||Image 6|
|Image 5||Image 1||Image 3||Image 5|
If the pool has an overlap, the next task is distributed only when the previous task is completed by the necessary number of Tolokers.
Use this parameter to:
Speed up collection of responses for majority vote check.
Assign tasks by priority.
Put important tasks in the beginning of the file. They will be completed faster and with the necessary overlap.
Learn more about how to set up a pool:
- Toloka API: Editing pool
- Toloka API: Changing pool priority
- Toloka-Kit: Editing pool
- Toloka-Kit: Patching pool
- Toloka-Kit recipe: Change pool priority
Try completing the tasks yourself. Ask your colleagues and friends to complete them. Find out average completion time and add 50% to it.
Overlap defines how many Tolokers complete the same pool task.
The best overlap is an overlap that provides satisfying quality of results. For most tasks that are not reviewed, overlap from “3” to “5” is enough. If the tasks are simple, overlap of “3” is likely to be enough. For tasks that are reviewed, set overlap to “1”.
Yes. Open edit mode for the pool and set a new overlap value. You don't need to restart the pool. Updating the settings is usually fast, but if there are many tasks, it may take several minutes.
The number of tasks depends on how difficult and time-consuming the tasks are. Keep the size reasonably small. Large task suites are unpopular, partly because they are inconvenient for Tolokers (for example, if the internet connection is unstable).
You've stopped the main pool. This could limit the number of Tolokers with access to the pool. Start the training pool again. There will be more Tolokers who can access the tasks.
The filters you set are too strict. For example, a strong restriction on a certain skill that most Tolokers don't have.
Too many Tolokers are banned. Ease the quality control rules.
If you change the time allocated for a task, the time value will apply to the tasks that have not yet been taken by the Tolokers. The same applies to the case when you close the pool. A Toloker who has an assignment in the active status can complete the assignment.
If you uploaded tasks to the pool using “smart mixing”, you can stop the pool and mark up your tasks: edit answers, hints, or delete tasks.
If you uploaded them using a different method, clone your pool and upload the new file with the corrected list of data to be labeled.