Ways to group tasks in suites

There are several ways to combine uploaded tasks into suites. You can choose the way that works best for you and specify additional settings.

Why this is important

Set manually

Enter the number of tasks per suite.

Smart mixing

Specify how many tasks of each type should be in each task suite. For example, 8 general tasks, 1 training and 1 control task. If necessary, specify the minimum number of tasks for each type in pool settings. The order of tasks depends on the Keep task order from uploaded data pool setting.

When to use

This method is useful if the created pool:

Sample settings

Features

  • Tasks are divided into lists: regular, control, and training.

  • The number of tasks of the given type that you specified in the settings is added from each list. By default, tasks are randomly selected.

  • Tasks in the suite are mixed up before the page is shown to the Toloker.

  • If there aren't enough general tasks and the Allow partial task suites option is set, the Toloker is given an incomplete task suite. Please note that the number of control and training tasks in this case must be complete.

Alert

If you upload a file via Smart mixing, you won't be able to use other ways of task distribution on the pages in this pool.

After uploading the tasks with smart mixing you will be able to mark up tasks and set selective majority vote checking.

How to distribute tasks as suites

Characteristics/upload typeSet manuallySmart mixing
To generate task suites, tasks are taken in the order of rows (from top to bottom) in an uploaded fileYesNo
Tasks are mixed within a suiteNoYes
Task suites are distributed to Tolokers in the same orderNoYes
Within identical task suites, control tasks are the same for all TolokersYesNo
Note

Please note that the above table is correct if you have more than one task per task suite. When you choose to have only one task in a task suite, the platform will sort not tasks but task suites. In this case, the Set manually option will not keep the order of the tasks and, on the contrary, the Smart mixing option will preserve the task order.

Control tasks

How many control tasks do I need to add?

We recommend adding at least 1% of control tasks in the pool. And for small pools — 5–10%.

Why's that?

If there are few control tasks in the open pool, add new control tasks.

What for
Note

To filter out Tolokers, use the Control tasks quality control rule. To rank Tolokers by the quality of control task responses, use a skill.

Setting overlap

If you upload tasks from the Toloka interface, infinite overlap is set automatically for control and training tasks, so that there is enough to mark up all general tasks.

You can set the overlap via the Toloka API or use Set manually for task allocation.

Important

Set infinite overlap for control tasks.

If another overlap value is set, control tasks may end during labeling and the pool will stop being labeled.

Tips and recommendations

  • If you used Set manually, you can find out the number of tasks per suite in the pool settings. But some suites may be incomplete.

  • If you uploaded tasks in a different way, you can check how they're grouped into suites in the Toloka interface for requesters. To do this, on the pool page, click filesDownload all tasks. You can use the Toloka API to check task distribution across suites.

See also

For developers

Troubleshooting

How do I specify smart mixing settings in the interface when uploading a file?
Can I set “smart mixing” after I uploaded tasks to a pool without it?
What is the maximum number of tasks per suite?
The same task appeared on different pages
How do I upload the file with the accepted assignments back to Toloka for projects with manual review? Where do I find the format of the upload data?
Contact support

Last updated: July 7, 2023

Introduction
Getting started
Important tips
Useful recommendations
Working with Toloka
Projects
Pools
Tasks
Results
Project analysis
Toloka settings
Task interface
Template Builder
HTML/CSS/JS editor
Help and support
FAQTroubleshootingSupportGlossary