Toloka documentation

Creating a pool

If you already have a pool and you want to create an identical one, clone the existing pool. If not, create a new pool.

New pool

A pool contains the settings of a set of task that are sent out for completion. The task interface is described in the project.

Note

You can first test the pool settings in the sandbox and then move them to the Toloka production version along with the project.

To create a pool:

  1. Click Add a pool on the project page.

  2. Set the Pool name (visible only to you) field. Only you will see this pool name on the project page.

  3. Specify the pool description which will be displayed instead of the project description in the task list for Tolokers. By default, Tolokers see the description from the project settings. To use a different description, uncheck the Use project description box and set Public description. If necessary, click + Private comment to add a private project description that only you will see.

  4. Click Create.

  5. Follow the steps in the pool wizard:

    Note

    After each step click Save to apply changes or Cancel to discard them.

    1. Select the audience for your task

      Select Tolokers using filters. Tasks will only be shown to matching Tolokers, rather than to all of them.

      Click the Add filter button and set the filter parameters.

    2. Setup quality control

      Quality control lets you get more accurate responses and restrict access to tasks for cheating users. Quality control consists of rules. All rules work independently.

      To learn more, see the Quality control overview section.

    3. Set the task price and overlap

      In the Price per task suite, $ field, set the amount of money to pay per task suite done by one Toloker.

      Pay attention to the Toloker interest at this price indicator. It shows how your price meets expectations of Tolokers. If the interest is low, you can raise price per task suite to have a bigger number of Tolokers in the project. If the interest is high, you can reduce price to save money.

      In the Overlap field, define how many Tolokers must do each task.

      You can also set up dynamic pricing and dynamic overlap.

    4. Prepare and upload data

      Upload tasks to the pool. This step will be enabled after you complete the previous steps.

    5. Add control tasks for checking performance

      Add control tasks to monitor the quality of task completion:

      1. Enter correct responses.

      2. Select the fields to use.

      3. Click Save and go to next or Skip task.

      4. Click Save all control tasks.

      Note

      This step will be enabled after you complete the previous steps. You can skip this step by clicking Do it later.

    6. Double-check your project and try out tasks

      At this step, check how the task will look from the Toloker's point of view.

      Note

      This step will be enabled after you complete the previous steps. You can skip this step by clicking Do it later.

    7. Add optional pool settings

      Set up advanced pool settings.

      Note

      This step will be enabled after you complete the previous steps. You can skip this step by clicking Use default settings.

After all the steps, you'll see the Set up is finished and your pool is ready for labeling tip on the pool page.

Tasks in pools will automatically be available in the web version of Toloka and the mobile app. If you want to change the default settings and limit the visibility of the task for any of the versions, add the Client filter and select the desired value: Toloka web version or Toloka for mobile.

Pool parameters

Parameter

Overview

Found at the step

My tasks may contain shocking or pornographic content

Whether the tasks have some content which can be marked as adult, including shocking or pornographic content. Tasks with such content are only given to Tolokers who agree to complete these types of tasks.

If you aren't sure whether the tasks have such content, enable this option.

Select the audience for your tasks

Filters

By setting the filters, you choose Tolokers that meet your requirements to complete your task. There are filters by various criteria, and filters by skills that you can create yourself.

You can filter Tolokers by language, country, city, and so on. Learn more

If you give training tasks to Tolokers, you can add a skill to the pool to choose Tolokers who already completed training. Learn more about skills

Select the audience for your tasks

Speed/quality balance

A setting for choosing Tolokers for your tasks. If you want to get responses very quickly, the quality may be doubtful. If you want very high quality, be prepared to wait longer. Find a balance that meets your needs. Learn more.

Select the audience for your tasks

Review task responses manually

Turn on this option to review the completed tasks manually.

Set the time period when you can check the tasks and reject them if the results are poor.

To learn more about manual review, see the Manual review page.

Set up quality control

Training

This field is only available if you have training pools in the project.

Training pool that will be linked to the main one.

Set up quality control

Level required, %

This field is only available if you have filled in the Training field.

Percentage of correct responses in training tasks (from 0 to 100) required to be admitted to the pool tasks. The calculation is based on the first response the Toloker gave in each task.

The minimum required level that you can set is 5. Tolokers who complete training with apercentage below this level won't have access to tasks.

Set up quality control

Review period in days

The number of days for reviewing and accepting tasks (max: 21). The Toloker will see the deadline for checking the tasks:

  • In the task information on the Toloka main page.
  • In the history of completed tasks.

Set up quality control

Price per task suite, $

Payment per task suite in US dollars. For cents, use the dot (".") as a separator. If the Pool type is General tasks, the minimum price per task suite is $0.005. For other pool types, you can set the price to zero. To learn more about price management, go to Setting up pricing.

Set the task price and overlap

Overlap

The number of Tolokers who should complete each task in the pool.

Configure this parameter if you need several people to complete your task. Overlap is necessary in tasks for collecting photo datasets, recording audio, or conducting surveys. In other types of tasks, it can improve the quality of results.

To save money, you can set up dynamic overlap (also known as incremental relabeling or IRL).

Dynamic overlap will help you save money. Learn more.

Set the task price and overlap

Dynamic pricing

Click Show dynamic pricing & overlap to go to the parameter. This parameter allows you to vary the task price depending on the Toloker's skill level. Learn more

Set the task price and overlap

Dynamic overlap

Click Show dynamic pricing & overlap to go to the parameter. You can use this parameter if you specify fixed values for the input data fields. It lets you save your budget and avoid setting fixed overlap for all pool tasks.

You set the range, the service analyzes the responses, their consistency, the level of Tolokers' skills and, if necessary, increases the overlap within the range you specified.

Set the task price and overlap

Pool priority within the project

Number from 0 to 100. Allows you to rank a pool within a project. First, a Toloker is assigned tasks from a pool with higher priority.

Add optional pool settings

Time per task suite, sec

The time allowed for completing a task suite, in seconds. Uncompleted tasks are redistributed to other Tolokers.

We recommend giving at least 60 seconds per task suite (including the time for page loading and sending responses).

If you set the interval too long, the tasks will stay open for a long time. If it is too short, the tasks won't be completed and paid for.

Add optional pool settings

Keep pool open after completion, sec

The interval during which the pool will remain open from the moment all tasks are completed. Minimum — 0, maximum — 259200 seconds (three days).

For example, you can use this parameter if you need an open pool where you regularly upload new tasks.

Additional settings

Pool closing date

The date the pool closes. The pool closes automatically on this date, even if the tasks aren't marked up.

Add optional pool settings

Pool type

Specify the pool type:

  • General tasks: The main pool that contains your regular tasks.

  • Exam: A pool that contains only control tasks.

  • Training: A main pool that consists only of training tasks and can have zero cost. Use it to train Tolokers before performing real tasks if the training pool doesn't work well for your project.

  • Retry: A pool that helps Tolokers who make mistakes improve their skills and get a second chance to complete tasks.

  • Other: If none of the previous options fit.

Note

If the price per task suite is zero, you must select the pool type.

Add optional pool settings

Keep task order

Option disabled (default value)

The uploaded tasks are grouped in pages and given to Tolokers in random order. Within the page, the task order is always random.

Example

If you specified 2 tasks per page when uploading the TSV file with image links to the pool, the system can generate them as follows:

Tasks in the file

Page 1

Page 2

Page 3

Image 1

Image 2

Image 6

Image 4

Image 2

Image 5

Image 1

Image 3

Image 3

Image 4

Image 5

Image 6

Option enabled

Tasks will be grouped on pages in the order they are listed in the task file.

Example

If you specified 2 tasks per page when uploading the file with image links to the pool, you're equally likely to get pages where the first link goes first and the second goes second, and the other way around:

Tasks in the file

Page 1

Page 2

Page 3

Image 1

Image 1

Image 3

Image 5

Image 2

Image 2

Image 4

Image 6

Image 3

or

or

or

Image 4

Image 2

Image 4

Image 6

Image 5

Image 1

Image 3

Image 5

Image 6

If the pool has an overlap, the next task is distributed only when the previous task is completed by the necessary number of Tolokers.

Use this parameter to:

  • Speed up collection of responses for majority vote check.

  • Issue tasks by priority.

    Put important tasks in the beginning of the file. They will be completed faster and with the necessary overlap.

Add optional pool settings

Metafields

To pass an array of metafields, specify key and value for each metafield.

Add optional pool settings

How Tolokers see pools

Tolokers can't see pools inside the project and select specific tasks (except for field tasks, where they can select points on the map). They also can't see pool settings, the number of tasks, or completion progress.

If at least one pool from the project is available to the Toloker, then this project is displayed on the main page as a card with the name, description, and price for the task. After selecting a project, the Toloker is issued a task from the pool. When they complete it, another one is issued, and so on.

A single project card is displayed if the pools in it differ only by name, quality control, or filters. In the latter case, the project card only includes tasks from the pools that the Toloker has access to.

Tolokers see one project card

For example, you create "Pool 1" and "Pool 2" with the same settings in the "Are there traffic signs in the photo?" project. If the Toloker has access to both pools, they are displayed as a single card on the main page.

Several cards are displayed if the pools differ in these settings:

  • Acceptance type
  • Public description
  • Task price

This is useful if you want to combine tasks in pools by subject, duration of audio recordings, or difficulty of evaluation.

Tolokers see several project cards

In this example, the Toloker has access to two pools with different prices and descriptions, so there are two project cards on the main page. The Toloker can choose any of them — for example, the one with the higher price.

Cloning a pool

To clone a pool, click → Clone at the top of the pool page or in the list of pools on the project page.

If you need to change the pool settings, open edit mode.

What's next

For developers

Troubleshooting

What overlap should I set?

Overlap defines how many Tolokers complete the same pool task.

The best overlap is an overlap that provides satisfying quality of results. For most tasks that are not reviewed, overlap from “3” to “5” is enough. If the tasks are simple, overlap of “3” is likely to be enough. For tasks that are reviewed, set overlap to “1”.

Can I change overlap after the pool is started?

Yes. Open edit mode for the pool and set a new overlap value. You don't need to restart the pool. Updating the settings is usually fast, but if there are many tasks, it may take several minutes.

What is the right time limit for the task completion?

Try completing the tasks yourself. Ask your colleagues and friends to complete them. Find out average completion time and add 50% to it.

How many tasks should be in a suite?

The number of tasks depends on how difficult and time-consuming the tasks are. Keep the size reasonably small. Large task suites are unpopular, partly because they are inconvenient for Tolokers (for example, if the internet connection is unstable).

Should I create a skill for every pool?

It is better to use one skill in a project. You can choose the way to calculate the skill:

  • Calculate the skill for each pool separately. The current skill value is the value of the skill in the pool the Toloker completed last. This option is convenient if:

    • The pools are intended for different groups of Tolokers (for example, there are filters by city or country).

    • Pools are started one by one and you don't want to take into account the responses in the previous pools to calculate the skill in the current pool.

    This calculation method is used by default when adding a quality control rule to a pool. For the control tasks block, leave the Recent control task responses to use field empty.

  • Calculate skill based on all tasks in a project This option is good if the pools are small and you don't need to have skill calculated for each pool.

    This option is available only for skills on control tasks. To use it, fill in the Recent control task responses to use field in pool quality control rules.