There are several ways to combine uploaded tasks into suites. You can choose the way that works best for you and specify additional settings.
Enter the number of tasks per suite.
Specify how many tasks of each type should be in each task suite. For example, 8 general tasks, 1 training and 1 control task. If necessary, specify the minimum number of tasks for each type in pool settings. The order of tasks depends on the Keep task order from uploaded data pool setting.
This method is useful if the created pool:
Contains control or training tasks in addition to the general tasks.
Has dynamic overlap (incremental relabeling, IRL) enabled.
Tasks are divided into lists: regular, control, and training.
The number of tasks of the given type that you specified in the settings is added from each list. By default, tasks are randomly selected.
Tasks in the suite are mixed up before the page is shown to the Toloker.
If there aren't enough general tasks and the Allow partial task suites option is set, the Toloker is given an incomplete task suite. Please note that the number of control and training tasks in this case must be complete.
If you upload a file via Smart mixing, you won't be able to use other ways of task distribution on the pages in this pool.
After uploading the tasks with smart mixing you will be able to mark up tasks and set selective majority vote checking.
| Characteristics/upload type | Set manually | Smart mixing |
|---|---|---|
| To generate task suites, tasks are taken in the order of rows (from top to bottom) in an uploaded file | Yes | No |
| Tasks are mixed within a suite | No | Yes |
| Task suites are distributed to Tolokers in the same order | No | Yes |
| Within identical task suites, control tasks are the same for all Tolokers | Yes | No |
Please note that the above table is correct if you have more than one task per task suite. When you choose to have only one task in a task suite, the platform will sort not tasks but task suites. In this case, the Set manually option will not keep the order of the tasks and, on the contrary, the Smart mixing option will preserve the task order.
We recommend adding at least 1% of control tasks in the pool. And for small pools — 5–10%.
If there are few control tasks in the open pool, add new control tasks.
To filter out Tolokers, use the Control tasks quality control rule. To rank Tolokers by the quality of control task responses, use a skill.
If you upload tasks from the Toloka interface, infinite overlap is set automatically for control and training tasks, so that there is enough to mark up all general tasks.
You can set the overlap via the Toloka API or use Set manually for task allocation.
Set infinite overlap for control tasks.
If another overlap value is set, control tasks may end during labeling and the pool will stop being labeled.
If you used Set manually, you can find out the number of tasks per suite in the pool settings. But some suites may be incomplete.
If you uploaded tasks in a different way, you can check how they're grouped into suites in the Toloka interface for requesters. To do this, on the pool page, click files → Download all tasks. You can use the Toloka API to check task distribution across suites.
Last updated: July 7, 2023