Pool statistics

Troubleshooting

You can track task completion on the pool page or by clicking Statistics. You can also get notifications by email, in the browser or on the Messages page. In the Profile, open the Notifications tab and choose which notifications you want to receive and how you want to receive them.


Field	Overview
Approximate finish time	The anticipated time for finishing tasks, in days and hours. Calculated using the average time per task suite. Formula:, where: is the number of task suite remaining uncompleted or those completed with insufficient overlap. is the average time per task suite. Calculated based on the amount of time since the pool opened and the number of task suites completed. The time is updated as task suites are completed by performers. Recommendations If the time is too long, most likely there are not enough users left in your project. Check the number of active performers and the average time per task suite. If the time has increased since the project started, you may have too many banned performers. If the time started out long and it hasn't decreased, and the number of active performers is small, check the filter settings and training. Perhaps few people can see your tasks or performers can't pass training.
Average submit time	The average time for completing a task suite in the pool. Specified in seconds. Calculated based on 1000 recently completed task suites. Recommendations If the average time is longer than you expected, and the quality of results is low, recheck the instructions. Perhaps your tasks are too complicated. Use decomposition and check the task interface. Then try testing your task on the same performers in the training or survey format. If the average time is short, and the quality of results is low, review quality control settings.
Expired tasks	The number of expired task suites. Includes the pages that performers failed to complete on time or decided to skip. You specify the time allotted for a task suite when setting up a pool. When the performer decides not to do the task (clicks Exit), the task gets the expired status. In this case, the performer can return to the task again, unless it was completed by someone else or the requester deleted it from the pool. Recommendations Performers might have skipped the task suite because: Tasks are too complex. There are too many tasks on the page. Tasks don't work and performers can't submit the assignments. Performers couldn't understand the instructions and task interface. If you have a lot of expired task suites in your pool, we recommend that you read ourTips for designing tasks.
Skipped tasks	The number of task suites skipped by performers. Includes the pages that the performers decided to skip and moved on to the next tasks. When the performer clicks Skip, the task gets the skipped status. In this case, the performer can no longer return to the task. Recommendations If you have a lot of skipped task suites in your pool, we recommend that you read our Tips for designing tasks and set up the skipped assignments rule.
Assignments	The graph shows the number of tasks by type. Submitted — The number of task suites submitted for review. Accepted — The number of accepted task suites. Skipped — The number of task suites skipped by performers. Expired — The number of expired task suites. Includes the pages that performers failed to complete on time or decided to skip. Recommendations If you have a lot of skipped or expired tasks, we recommend that you read our Tips for designing tasks and set up the skipped assignments rule.
Average overlap	The average number of performers who completed the same task in the pool. Recommendations If you think that this number is too large or small, you may have incorrectly configured Recompletion of assignments from banned users or Processing rejected and accepted tasks. You may also want to check: Dynamic overlap (incremental relabeling, IRL). Random majority vote check.
Assignment completion time	The average time spent by performers to complete a task suite in the pool. Specified in seconds.
Budget spent (+ markup)	The amount of money spent in the pool. The fee amount is shown in parentheses. If you spent more than you planned, check the price settings in the pool and the quality control rules. For example, Dynamic pricing and Recompletion of assignments from banned users increase the cost of the pool. Please note that the minimum fee is $0.005. Recommendations First, check the quality control rules, disable the rules that increase overlap, and then revise the price per task suite. You might need to reduce it. but don't make it unreasonably small. You can also increase or decrease the number of tasks per suite and adjust the price accordingly.
Estimated budget (+ markup)	The amount of money spent and expected costs (if all the tasks are completed and all responses are approved). The fee amount is shown in parentheses. Recommendations This indicator uses available data to predict the total cost. If you are not satisfied with it, stop the pool and change the settings. Think about whether you need dynamic overlap, dynamic pricing, and, for example, recompletion of assignments from banned users. Review the price per task suite. You might be able to lower it without losing performers.
Spending (excluding markup)	The graph shows the amount of money spent excluding markup. Spending on bonuses — The amount of money spent on extra rewards. Spending on assignments — The amount of money spent on tasks.
Average task price	The average price in USD per general task in the pool, including overlap. Note that this is the price per task, not per suite with overlap. Recommendations If you think that the price is too high, check the price settings in the pool and the quality control rules. For example, Dynamic pricing and Recompletion of assignments from banned users increase the cost of the pool.
	This graph shows the percentage of correct responses in the control and training tasks. Recommendations If the percentage is too low, maybe your tasks are too complicated or the instructions are not clear enough. Maybe there are errors in the control tasks or they are irrelevant. If the tasks have links, make sure that they work. Another possible reason is incorrect settings for the quality control rules. Maybe they fail to filter negligent performers.
Blocked by rules	The number of performers banned by the quality control rules. Note that the number of performers is counted separately for each rule. Recommendations If the number is too large or too small, check the settings for the quality control rules.
Banned users	The total number of performers banned in this pool. The graph shows two indicators: In the pool — Number of performers banned in the pool. By the requester — Number of performers banned by the requester. Note that the graph only shows banned performers. It doesn't include performers whose access is suspended. Recommendations If there are too many banned users, check the quality control rules. Maybe they are too strict.
Active users having access to pool	The number of performers selected for the pool with filters. The number includes only performers who viewed and completed tasks in Toloka in the last hour. Recommendations If this number is too small, check the filter settings and the quality control rules.
Interested in pool	The number of performers who started or completed at least one task suite. It includes both interested and engaged performers. Recommendations If there is a large difference between the numbers of interested and engaged performers, perhaps the instructions are unclear, the task is too complicated, or there are errors in the interface.
Submitted in pool	The number of performers who completed at least one task suite. Recommendations If this number is low but you're satisfied with the number of properly completed tasks, it means that you set up the project correctly.
Submitted assignments per 1 user	The average number of task suites per performer. If your pool is large and it is taking a long time to finish it, but there are very few task suites per performer, you may have incorrectly configured quality control rules or filters. Recommendations If the number of task suites per performer is close to the threshold for triggering quality control rules, check the quality control rules that are responsible for banning users. You should also check the banned users and banned by rules graphs.
Performers completing tasks in the pool	This graph shows the total number of performers who completed at least one task suite in the pool. Recommendations If there aren't enough performers, check the settings for the quality control rules and filters. You should also check the banned users and banned by rules graphs.


Field	Overview
Approximate finish time	The anticipated time for finishing tasks, in days and hours. Calculated using the average time per task suite. Formula:, where: is the number of task suite remaining uncompleted or those completed with insufficient overlap. is the average time per task suite. Calculated based on the amount of time since the pool opened and the number of task suites completed. The time is updated as task suites are completed by performers. Recommendations If the time is too long, most likely there are not enough users left in your project. Check the number of active performers and the average time per task suite. If the time has increased since the project started, you may have too many banned performers. If the time started out long and it hasn't decreased, and the number of active performers is small, check the filter settings and training. Perhaps few people can see your tasks or performers can't pass training.
Average submit time	The average time for completing a task suite in the pool. Specified in seconds. Calculated based on 1000 recently completed task suites. Recommendations If the average time is longer than you expected, and the quality of results is low, recheck the instructions. Perhaps your tasks are too complicated. Use decomposition and check the task interface. Then try testing your task on the same performers in the training or survey format. If the average time is short, and the quality of results is low, review quality control settings.
Expired tasks	The number of expired task suites. Includes the pages that performers failed to complete on time or decided to skip. You specify the time allotted for a task suite when setting up a pool. When the performer decides not to do the task (clicks Exit), the task gets the expired status. In this case, the performer can return to the task again, unless it was completed by someone else or the requester deleted it from the pool. Recommendations Performers might have skipped the task suite because: Tasks are too complex. There are too many tasks on the page. Tasks don't work and performers can't submit the assignments. Performers couldn't understand the instructions and task interface. If you have a lot of expired task suites in your pool, we recommend that you read ourTips for designing tasks.
Skipped tasks	The number of task suites skipped by performers. Includes the pages that the performers decided to skip and moved on to the next tasks. When the performer clicks Skip, the task gets the skipped status. In this case, the performer can no longer return to the task. Recommendations If you have a lot of skipped task suites in your pool, we recommend that you read our Tips for designing tasks and set up the skipped assignments rule.
Assignments	The graph shows the number of tasks by type. Submitted — The number of task suites submitted for review. Accepted — The number of accepted task suites. Skipped — The number of task suites skipped by performers. Expired — The number of expired task suites. Includes the pages that performers failed to complete on time or decided to skip. Recommendations If you have a lot of skipped or expired tasks, we recommend that you read our Tips for designing tasks and set up the skipped assignments rule.
Average overlap	The average number of performers who completed the same task in the pool. Recommendations If you think that this number is too large or small, you may have incorrectly configured Recompletion of assignments from banned users or Processing rejected and accepted tasks. You may also want to check: Dynamic overlap (incremental relabeling, IRL). Random majority vote check.
Assignment completion time	The average time spent by performers to complete a task suite in the pool. Specified in seconds.
Budget spent (+ markup)	The amount of money spent in the pool. The fee amount is shown in parentheses. If you spent more than you planned, check the price settings in the pool and the quality control rules. For example, Dynamic pricing and Recompletion of assignments from banned users increase the cost of the pool. Please note that the minimum fee is $0.005. Recommendations First, check the quality control rules, disable the rules that increase overlap, and then revise the price per task suite. You might need to reduce it. but don't make it unreasonably small. You can also increase or decrease the number of tasks per suite and adjust the price accordingly.
Estimated budget (+ markup)	The amount of money spent and expected costs (if all the tasks are completed and all responses are approved). The fee amount is shown in parentheses. Recommendations This indicator uses available data to predict the total cost. If you are not satisfied with it, stop the pool and change the settings. Think about whether you need dynamic overlap, dynamic pricing, and, for example, recompletion of assignments from banned users. Review the price per task suite. You might be able to lower it without losing performers.
Spending (excluding markup)	The graph shows the amount of money spent excluding markup. Spending on bonuses — The amount of money spent on extra rewards. Spending on assignments — The amount of money spent on tasks.
Average task price	The average price in USD per general task in the pool, including overlap. Note that this is the price per task, not per suite with overlap. Recommendations If you think that the price is too high, check the price settings in the pool and the quality control rules. For example, Dynamic pricing and Recompletion of assignments from banned users increase the cost of the pool.
	This graph shows the percentage of correct responses in the control and training tasks. Recommendations If the percentage is too low, maybe your tasks are too complicated or the instructions are not clear enough. Maybe there are errors in the control tasks or they are irrelevant. If the tasks have links, make sure that they work. Another possible reason is incorrect settings for the quality control rules. Maybe they fail to filter negligent performers.
Blocked by rules	The number of performers banned by the quality control rules. Note that the number of performers is counted separately for each rule. Recommendations If the number is too large or too small, check the settings for the quality control rules.
Banned users	The total number of performers banned in this pool. The graph shows two indicators: In the pool — Number of performers banned in the pool. By the requester — Number of performers banned by the requester. Note that the graph only shows banned performers. It doesn't include performers whose access is suspended. Recommendations If there are too many banned users, check the quality control rules. Maybe they are too strict.
Active users having access to pool	The number of performers selected for the pool with filters. The number includes only performers who viewed and completed tasks in Toloka in the last hour. Recommendations If this number is too small, check the filter settings and the quality control rules.
Interested in pool	The number of performers who started or completed at least one task suite. It includes both interested and engaged performers. Recommendations If there is a large difference between the numbers of interested and engaged performers, perhaps the instructions are unclear, the task is too complicated, or there are errors in the interface.
Submitted in pool	The number of performers who completed at least one task suite. Recommendations If this number is low but you're satisfied with the number of properly completed tasks, it means that you set up the project correctly.
Submitted assignments per 1 user	The average number of task suites per performer. If your pool is large and it is taking a long time to finish it, but there are very few task suites per performer, you may have incorrectly configured quality control rules or filters. Recommendations If the number of task suites per performer is close to the threshold for triggering quality control rules, check the quality control rules that are responsible for banning users. You should also check the banned users and banned by rules graphs.
Performers completing tasks in the pool	This graph shows the total number of performers who completed at least one task suite in the pool. Recommendations If there aren't enough performers, check the settings for the quality control rules and filters. You should also check the banned users and banned by rules graphs.

Troubleshooting

Why has the speed of pool completion dropped?

Possible reasons:

You've stopped the main pool. This could limit the number of performers with access to the pool. Start the training pool again. There will be more performers who can access the tasks.
The filters you set are too strict. For example, a strong restriction on a certain skill that most users don't have.
Too many users are banned. Ease the quality control rules.

How can I speed up the pool completion?

To motivate performers, assign a public skill and use dynamic pricing.
Try to increase the project rating, so that your task is higher in the list of tasks for performers.
Adjust the quality-speed ratio.
Set a higher priority for the pool among other project pools.

Why is the maximum number of submitted assignments in the progress bar less than the total number of uploaded tasks?

The progress bar shows the number of task suites including the overlap. If the overlap is greater than one, the number of task suites is different from the total number of tasks.