Adding a training

With training pools, Tolokers can practice before getting started. You can provide access to the general task pool only to those Tolokers who passed the training.

The training pool contains only training tasks. Tasks in the training pool aren't paid for.

Note. If the training pool functionality doesn't meet your needs, create a main pool and set the pool type to Training .

How does it work?

  1. You create a training pool. A training skill is created automatically. Newly created training pools use a skill that was already created.
  2. You upload tasks to the pool and mark them up. You can upload tasks that are already marked up.
  3. You link the training to the main pool.
  4. You run both pools in turn.
  5. The Toloker sees only the training tasks and decides to complete them.

    If the Toloker makes a mistake, they are shown a hint. To go to the next assignment, they need to answer all questions correctly. If the tips aren't helpful, the Toloker won't be able to complete the training.

  6. When the Toloker completes the training, they are assigned a training skill. The skill value is the percentage of correct responses submitted by the Toloker.

    Only the first responses of the Toloker are taken into account. Therefore, the correct responses in the hints don't affect the the training skill value.
  7. If the percentage of correct responses is higher or equal to the Level required value in the linked pool, the Toloker will have access to this pool.

    Pools that aren't linked to the training pool will be available to everyone unless you set the filters. Regardless of whether the Toloker passed the training or not.
    Tip. You can link multiple pools to one training pool and set different required levels for them. This lets you assign more difficult tasks to more advanced Tolokers.

How do I create a training pool?

  1. Open the project page.

  2. Go to the Training tab.

  3. Click the Add training button.

  4. Fill in the training settings fields.

    You can use the Retry after field to set up repeated training.
  5. Click Save training.

After you created a training pool

  1. Upload tasks to the pool and mark them up (add correct responses and hints). All tasks must be training. You can upload tasks that are already marked up.

    What should be a good hint?

    Avoid wordings like: “You answered incorrectly, please provide the correct response”. The Toloker learns when the hint explains the essence of their mistake.

    Make the hints clear. Explain which response should be chosen and why.

  2. Link the training to the main pool. For this, edit the main pool by filling in the fields:
    • Training — Name of the training pool.
    • Level required — Percentage of correct responses in training tasks (from 5 to 100) required to be admitted to the pool tasks. The calculation is based on the first response the Toloker gave in each task.

Training settings

Field

Overview

Guidelines

Instructions for the training tasks.

By default, the project instructions are displayed. To write separate instructions for training, deselect the Use project instructions box.

Training title

Name of the training pool (not visible to the Toloker).

Adult content

Whether the training tasks have porn content.

Time on task

The time allowed for completing a task suite, in seconds.

Retry after

The number of days after which the Toloker can access the training again. If not specified, training skill is issued for an indefinite time, and the value is fixed.

Learn more about how it works.

Issue in task uploading order If this option is enabled, tasks are assigned to the Toloker in the order they are listed in the file with tasks.
Shuffle on page If this option is enabled, tasks on the page are shown to the Toloker in random order.
Complete passing

If this option is enabled, the Toloker must complete all the tasks in this pool to pass the training.

You can load more training tasks than required for passing the training and specify the number of pages required for setting the skill and accessing the linked pools.
Required for passing The number of assignments the Toloker must complete to pass the training.

Field

Overview

Guidelines

Instructions for the training tasks.

By default, the project instructions are displayed. To write separate instructions for training, deselect the Use project instructions box.

Training title

Name of the training pool (not visible to the Toloker).

Adult content

Whether the training tasks have porn content.

Time on task

The time allowed for completing a task suite, in seconds.

Retry after

The number of days after which the Toloker can access the training again. If not specified, training skill is issued for an indefinite time, and the value is fixed.

Learn more about how it works.

Issue in task uploading order If this option is enabled, tasks are assigned to the Toloker in the order they are listed in the file with tasks.
Shuffle on page If this option is enabled, tasks on the page are shown to the Toloker in random order.
Complete passing

If this option is enabled, the Toloker must complete all the tasks in this pool to pass the training.

You can load more training tasks than required for passing the training and specify the number of pages required for setting the skill and accessing the linked pools.
Required for passing The number of assignments the Toloker must complete to pass the training.

Repeated training

After completing the training (successfully or not), the Toloker is assigned a training skill, and the value is fixed.

New training pools in the project aren't shown to the Toloker, as long as they have a training skill.

The lifespan of the skill and the need for retraining depends on what you specify in the Retry after field.

How does it work?
Repeated training isn't set up

The training skill is assigned for an indefinite time, so the training pools in this project are no longer available to the Toloker until you delete the training skill manually.

Repeated training

The training skill is deleted after the specified number of days if the Toloker didn't complete tasks in the linked pools.

This means that:
  • Those who failed to complete the training tasks can complete the training again after the specified number of days.
  • Those who didn't complete tasks for a long time have to repeat the training.
  • Those who actively perform tasks don't have to spend the time on retraining.
Note. Completing tasks in pools that aren't linked to the training pool isn't taken into account.
How do I set it up?
Send only inactive Tolokers for retraining

Repeated training isn't needed if the Toloker already completes tasks in the linked pools.

Enter the period in which the Toloker must complete at least one task in the linked pools to avoid retraining and keep the skill in the Retry after field.

Attention. Don't forget to link the main and training pools. Completing tasks in pools that aren't linked to the training pool isn't taken into account. The training skill will be lost.
Forbid repeated training

Leave the Retry after empty. Then the skill will be assigned to the Toloker for an indefinite time, and the value is fixed.

You can delete a skill or change the value manually.

Note. Tolokers will get access only to those pools for which their skill value is higher than the value in the Level required field.

How to archive a training pool

A pool with the “Archived” status can't be started or edited.

By default, archived training pools are not visible in the list of project trainings. To view them:
  1. Open the project page.
  2. In the Training tab, select Archived.

The training pool is automatically archived if no action is performed in it for 30 days.

To archive a training pool manually, click  → Archive at the top of the pool page or in the list of training options on the project page.

If these buttons don't work, make sure that all the main pools that the training pool is linked to are sent to the archive.

Troubleshooting

Setting up training
How do I precede my task with mandatory control questions to check that the Toloker understood my instructions? Would such training or control tasks be similar to the general tasks?

The training and control questions must meet your project specification. However, you can create a separate project with your instructions, survey, and sample videos. Then you can assign a skill to users based on their responses. You can use this skill to admit Tolokers to the main project.

How do I make one parameter mandatory and the other parameters optional in my training task?

In the task file, leave empty control values for the optional output data.

Why do I have an infinite number of pages in the training pool?

Tasks have infinite overlap in the training pool. As long as the training pool is open and the training is running, users can access the tasks. Learn more about training pools.

How do I insert a link in the GOLDEN field?

Text in the GOLDEN field must match the control text exactly.

Usually, if you copy site links from the browser, the copied links have the same format. But this is not the case when the link is trimmed or typed manually.

Check the links that you use. There are several ways to unify links:
  • Add requirements for the link format in your instructions and hints in your training pool.
  • Use RegExp in your JS to trim the received links and write the result to the new output field, and then match the received value against the control value.
If I upload tasks using smart mixing, does it mean that the same file should contain both the control tasks and general tasks, or can I upload them separately?

Smart mixing is set up when you upload tasks to the pool. After creating a pool, click Upload and select the method for generating task suites. You can upload them using separate files or one file, arranging them in any order.

Can I automatically pause accepting applications for the training pool if the necessary number of Tolokers have been trained and are already doing the tasks?

You can close the pool manually at any time using the interface. However, you can't set the number of users that should complete the training pool for it to close automatically.

How do I check that the Tolokers don't cheat during training?

Training helps users learn how to complete your task and figure out the instructions.

Based on the training results, you can select Tolokers who did well enough for the main pool.

However, the mere fact that a Toloker completes your training pool successfully doesn't guarantee that they will afterwards demonstrate high quality on your general tasks. Tolokers who show a high level of accuracy during the training could have obtained correct responses from others.

Besides the training, be sure to add quality control rules and control tasks to your main pools. This way you can ensure the quality throughout the task performance process.

If the task requires that the Tolokers send free-format responses or data files, use non-automatic acceptance to pay for tasks after they are reviewed.

Why does the training pool allow smart mixing but doesn't allow adding by empty row?

This is a technical limitation of training pools. You can only upload tasks to your training pools this way. If you want to upload tasks to the training pool suite-by-suite, create the main pool, set the pool type to Training, and set the price to zero.

How do I create two active training pools: the first one for practice and the second one to admit the Tolokers to the main pool?

Create the first pool based on the training pool and the second pool based on the main pool with the pool type set to Exam. If a pool contains only control and/or training tasks, the price can be set to zero.

In the exam pool, you can create a skill reflecting the exam result and granting admission to the main pool. For example, If the number of responses is ≥ 10, set the skill value in the <exam skill> as % of correct responses.

In your exam user requirements, specify: <exam skill> <80 or = is missing>.

In the main pool, set up a filter: <exam skill> >= 80 and <main skill> >= 70 or = Is missing>. You can choose the skill values depending on how well the Tolokers handle your task.

How do I create a training pool so that the Toloker might fail it but still be admitted to the general task pool?

Create a main pool of the Training type. Add only training tasks. To assign a skill, use the Control tasks rule. To allow access to the general tasks to users with any skill level, set up the following filter in the main pool: <skill> >= 0 or <skill> ≠ 0.

However, we don't advise giving access to general tasks to Tolokers who failed training.

How do I create a file with training tasks?
For training tasks, you need to:
  • Select the correct responses in the GOLDEN:result column.
  • Fill in the HINT:text column. It stores a hint to be shown if the Toloker selects an incorrect response option.
Do users have to complete all the tasks in the training pool?

If you enabled incomplete training and specified the number of training pages required, users don't have to fully complete the training in order to pass. If you didn't make these settings, the Tolokers have to complete all the tasks in the training pool to get a training skill.

Is the training considered an active pool when the main pool is closed?

Yes, it is.

How do I make the training optional so that Tolokers can decide themselves whether to take it or not?

Training is designed to select Tolokers for the general task. That's why training must be linked to the main pool and become inactive as soon as the main pool closes.

The Toloker is trained to get access to your paid tasks. If the training is optional, there probably won't be very many people who choose to complete it. Technically, “optional” training can be based on a main pool that includes some training tasks.

To show the training separately from other pools, disable Use project description and use this field to specify that this is an optional set of training tasks. In the pool settings, select the Training type.

Can I create training for projects where it is not possible to formulate the correct response exactly or review it automatically?

You can't create a training like this, because for the response to be counted as correct it must exactly match the control text.

For projects using free text input or attached files, you can make a pre-selection task with non-automatic acceptance. You can admit good Tolokers to your main pool based on their skill.

Can I implement non-automatic acceptance in the training pool?

You can't use non-automatic acceptance in your training pool.

However, you can create a training pool with the Training type based on your main pool and enable non-automatic acceptance there.

Training skill
More than 500 Tolokers passed the training, but the training skill shows only 30.

The pool shows the total number of Tolokers that completed at least one assignment. A training skill can be lost over time if you set repeated training in the pool settings. This setting allows a Toloker to pass the training again after a certain period if the Toloker didn't complete any tasks in associated pools or if there was a large time gap between completing tasks (for example, because of the ban). The training skill displays the Tolokers who either recently completed training, or regularly complete your tasks so that the skill doesn't expire.

More Tolokers were trained than the training skill shows

The pool shows the total number of Tolokers that completed at least one assignment. A training skill can be lost over time if you set repeated training in the pool settings. This setting allows a Toloker to pass the training again after a certain period if the Toloker didn't complete any tasks in associated pools or if there was a large time gap between completing tasks (for example, because of a ban). The training skill displays the Tolokers who either recently completed training, or regularly complete your tasks so that the skill doesn't expire.

Which parameter affects the skill expiration?

Validity period of the training skills is governed by the Retry after parameter.

The skill is deleted in the specified number of days if the Toloker:
  • Has a skill value lower than in the Level required field.
  • Didn't complete any tasks linked to the training during this period.
If their skill expires, your users need to complete the training again.
How do I know when a particular Toloker got the skill?
  1. Go to the Toloker card.
  2. Click the Profile tab.
  3. Find the required skill in the list and download the history of its changes.
Test and retry
What's the difference between the exam pool that I pay for and the main pool?

Exam is a pool that contains only the control tasks. Usually it's small and intended to check how users learned to do your tasks after they read the instructions and completed the training.

Unlike your main pool, you already know correct responses for every task. You can set the price to zero. Based on the results of responses to control tasks, you can assign a skill to the Tolokers and then specify it in the main pool as a filter. For example, ≥ 80 or ≠ Is missing. You don't have to create an exam, because the training pool provides enough practice for simple tasks. But many requesters also use exams.

How do I set up a retry pool for my project?

You can create a retry pool similarly to an exam pool. In the pool settings, select the type Retry. In the retry pool filters, specify the upper and lower values of the <main skill> that the Tolokers must get in order to be admitted to the retry pool.

For example, if the main pool admits users with a skill of 70 or higher, then you can route the people with a skill between 40 and 69 to the retry pool.

To get a valid “range”, enter the skill twice: with an upper and lower value. For example: <basic skill > <70 and main skill >=40.

We recommend that you don't make your exam and retry pools too lengthy, because Tolokers don't like to do zero-price tasks. 10-20 tasks is enough, depending on complexity.
How do I create an exam with a preset number of correct responses?

To do this, under Test result, go to Recent tasks to use and specify the number of recent responses from the Toloker.

Let's say you need to create an exam with three tasks, one task per suite. If the Toloker succeeds in two out of three tasks, they get the skill.

If your task uses assignment review (non-automatic acceptance), to set up such a rule you need to specify 3 for "Total reviewed responses". As you can see in the screenshot, in the first case, all the Tolokers who completed 3 assignments and whose answers are reviewed will get the skill. In the second case, only those who have 2 or 3 assignments accepted will get the skill.

How do I create a training and honey pots with an exam to get an output response other than the control value?

For a control or training assignment to be counted as correct, it must exactly match the control assignment. To do this, you need to normalize the response text using JavaScript: remove spaces, punctuation marks, special characters, and capital letters, and write the result in a separate output field. Now you can match the processed assignment text against your control text.

Another option for selecting Tolokers for a project of this type is assignment review (non-automatic acceptance).