Adding a training

With training pools, performers can practice before getting started. You can provide access to the main task pool only to those performers who passed the training.

The training pool contains only training tasks. Tasks in the training pool aren't paid for.

Note. If the training pool functionality doesn't meet your needs, create a main pool and set the pool type to Training .

How does it work?

  1. You create a training pool. A training skill is created automatically. Newly created training pools use a skill that was already created.
  2. You upload tasks to the pool and mark them up. You can upload tasks that are already marked up.
  3. You link the training to the main pool.
  4. You run both pools in turn.
  5. The performer sees only the training tasks and decides to complete them.

    If the performer makes a mistake, they are shown a hint. To go to the next assignment, they need to answer all questions correctly. If the tips aren't helpful, the performer won't be able to complete the training.

  6. When the performer completes the training, they are assigned a training skill. The skill value is the percentage of correct responses submitted by the performer.

    Only the first responses of the performer are taken into account. Therefore, the correct responses in the hints don't affect the the training skill value.
  7. If the percentage of correct responses is higher or equal to the Level required value in the linked pool, the performer will have access to this pool.

    Pools that aren't linked to the training pool will be available to everyone unless you set the filters. Regardless of whether the performer passed the training or not.
    Tip. You can link multiple pools to one training pool and set different required levels for them. This lets you assign more difficult tasks to more advanced performers.

How do I create a training pool?

  1. Open the project page.

  2. Go to the Training tab.

  3. Click the Add training button.

  4. Fill in the training settings fields.

    You can use the Retry after field to set up repeated training.
  5. Click Save training.

After you created a training pool

  1. Upload tasks to the pool and mark them up (add correct responses and hints). All tasks must be training. You can upload tasks that are already marked up.

    What makes a good hint?

    Avoid wordings like: “You answered incorrectly, please provide the correct response”. The performer learns when the hint explains the essence of their mistake.

    Make the hints clear. Explain which response should be chosen and why.

  2. Link the training to the main pool. For this, edit the main pool by filling in the fields:
    • Training — Name of the training pool.
    • Level required — Percentage of correct responses in training tasks (from 0 to 100) required to be admitted to the pool tasks. The calculation is based on the first response the performer gave in each task.

Training settings

Field

Overview

Guidelines

Instructions for the training tasks.

By default, the project instructions are displayed. To write separate instructions for training, deselect the Use project instructions box.

Training title

Name of the training pool (not visible to the performer).

Adult content

Whether the training tasks have porn content.

Time on task

The time allowed for completing a task suite, in seconds.

Retry after

The number of days after which the performer can access the training again. If not specified, training skill is issued for an indefinite time, and the value is fixed.

Learn more about how it works.

Issue in task uploading order If this option is enabled, tasks are assigned to the user in the order they are listed in the TSV file.
Shuffle on page If this option is enabled, tasks on the page are shown to the performer in random order.
Complete passing

If this option is enabled, the performer must complete all the tasks in this pool to pass the training.

You can load more training tasks than required for passing the training and specify the number of pages required for setting the skill and accessing the linked pools.
Required for passing The number of assignments the performer must complete to pass the training.

Field

Overview

Guidelines

Instructions for the training tasks.

By default, the project instructions are displayed. To write separate instructions for training, deselect the Use project instructions box.

Training title

Name of the training pool (not visible to the performer).

Adult content

Whether the training tasks have porn content.

Time on task

The time allowed for completing a task suite, in seconds.

Retry after

The number of days after which the performer can access the training again. If not specified, training skill is issued for an indefinite time, and the value is fixed.

Learn more about how it works.

Issue in task uploading order If this option is enabled, tasks are assigned to the user in the order they are listed in the TSV file.
Shuffle on page If this option is enabled, tasks on the page are shown to the performer in random order.
Complete passing

If this option is enabled, the performer must complete all the tasks in this pool to pass the training.

You can load more training tasks than required for passing the training and specify the number of pages required for setting the skill and accessing the linked pools.
Required for passing The number of assignments the performer must complete to pass the training.

Repeated training

After completing the training (successfully or not), the performer is assigned a training skill, and the value is fixed.

New training pools in the project aren't shown to the performer, as long as they have a training skill.

The lifespan of the skill and the need for retraining depends on what you specify in the Retry after field.

How does it work?
Repeated training isn't set up

The training skill is assigned for an indefinite time, so the training pools in this project are no longer available to the performer until you delete the training skill manually.

Repeated training

The training skill is deleted after the specified number of days if the performer didn't complete tasks in the linked pools.

This means that:
  • Those who failed to complete the training tasks can complete the training again after the specified number of days.
  • Those who didn't complete tasks for a long time have to repeat the training.
  • Those who actively perform tasks don't have to spend the time on retraining.
Note. Completing tasks in pools that aren't linked to the training pool isn't taken into account.
How do I set it up?
Send only inactive performers for retraining

Repeated training isn't needed if the performer already completes tasks in the linked pools.

Enter the period in which the performer must complete at least one task in the linked pools to avoid retraining and keep the skill in the Retry after field.

Attention. Don't forget to link the main and training pools. Completing tasks in pools that aren't linked to the training pool isn't taken into account. The training skill will be lost.
Forbid repeated training

Leave the Retry after empty. Then the skill will be assigned to the performer for an indefinite time, and the value is fixed.

You can delete a skill or change the value manually.

Note. Performers will get access only to those pools for which their skill value is higher than the value in the Level required field.

How to archive a training pool

A pool with the “Archived” status can't be started or edited.

By default, archived training pools are not visible in the list of project training options. To view them:
  1. Open the project page.
  2. In the Training tab, select Archived.

The training pool is automatically archived if no action is performed in it for 30 days.

To archive a training pool manually, click  → Archive at the top of the pool page or in the list of training options on the project page.

If these buttons don't work, make sure that all the main pools that the training pool is linked to are sent to the archive.

Troubleshooting

Setting up training
How do I precede my task with mandatory control questions to check that the user understood my instructions? Would such training or control tasks be similar to the main tasks?

The training and control questions must meet your project specification. However, you can create a separate project with your instructions, survey, and sample videos. Then you can assign a skill to users based on their responses. You can use this skill to admit performers to the main project.

How do I make one parameter mandatory and the other parameters optional in my training task?

In the task file, leave empty control values for the optional output data.

Why do I have an infinite number of pages in the training pool?

Tasks have infinite overlap in the training pool. As long as the training pool is open and the training is running, users can access the tasks. Learn more about training pools.

How do I insert a link in the GOLDEN field?

Text in the GOLDEN field must match the control text exactly.

Usually, if you copy site links from the browser, the copied links have the same format. But this is not the case when the link is trimmed or typed manually.

Check the links that you use. There are several ways to unify links:
  • Add requirements for the link format in your instructions and hints in your training pool.
  • Use RegExp in your JS to trim the received links and write the result to the new output field, and then match the received value against the control value.
How do I use smart mixing to upload my main tasks separately from control tasks?

Smart mixing is set up when you upload tasks to the pool. After creating a pool, click Upload and select the method for generating task suites. You can upload them using separate files or one file, arranging them in any order.

Can I automatically pause accepting applications for the training pool if the necessary number of performers have been trained and are already doing the tasks?

You can close the pool manually at any time using the interface. However, you can't set the number of users that should complete the training pool for it to close automatically.

How do I check that the performers don't cheat during training?

Training helps users learn how to complete your task and figure out the instructions.

Based on the training results, you can select performers who did well enough for the main pool.

However, the mere fact that a performer completes your training pool successfully doesn't guarantee that they will afterwards demonstrate high quality on your main tasks. Performers who show a high level of accuracy during the training could have obtained correct responses from others.

Besides the training, be sure to add quality control rules and control tasks to your main pools. This way you can ensure the quality throughout the task performance process.

If the task requires that the users send free-format responses or data files, use non-automatic acceptance to pay for tasks after they are reviewed.

Why does the training pool allow smart mixing but doesn't allow adding by empty row?

This is a technical limitation of training pools. You can only upload tasks to your training pools this way. If you want to upload tasks to the training pool suite-by-suite, create the main pool, set the pool type to Training, and set the price to zero.

How do I create two active training pools: the first one for practice and the second one to admit the users to the main pool?

Create the first pool based on the training pool and the second pool based on the main pool with the pool type set to Exam. If a pool contains only control and/or training tasks, the price can be set to zero.

In the exam pool, you can create a skill reflecting the exam result and granting admission to the main pool. For example, If the number of responses is ≥ 10, set the skill value in the <exam skill> as % of correct responses.

In your exam user requirements, specify: <exam skill> <80 or = is missing>.

In the main pool, set up a filter: <exam skill> >= 80 and <main skill> >= 70 or = Is missing>. You can choose the skill values depending on how well the performers handle your task.

How do I create a training pool so that the performer might fail it but still be admitted to the main task pool?

Create a main pool of the Training type. Add only training tasks. To assign a skill, use the Control tasks rule. To allow access to the main tasks to users with any skill level, set up the following filter in the main pool: <skill> >= 0 or <skill> ≠ 0.

However, we don't advise giving access to main tasks to performers who failed training.

How do I create a file with training tasks?
For training tasks, you need to:
  • Select the correct responses in the GOLDEN:result column.
  • Fill in the HINT:text column. It stores a hint to be shown if the user selects an incorrect response option.
Do users have to complete all the tasks in the training pool?

If you enabled incomplete training and specified the number of training pages required, users don't have to fully complete the training in order to pass. If you didn't make these settings, the users have to complete all the tasks in the training pool to get a training skill.

Is the training considered an active pool when the main pool is closed?

Yes, it is.

How do I make the training optional so that performers can decide themselves whether to take it or not?

Training is designed to select performers for the main task. That's why training must be linked to the main pool and become inactive as soon as the main pool closes.

The user is trained to get access to your paid tasks. If the training is optional, there probably won't be very many people who choose to complete it. Technically, “optional” training can be based on a main pool that includes some training tasks.

To show the training separately from other pools, disable Use project description and use this field to specify that this is an optional set of training tasks. In the pool settings, select the Training type.

Can I create training for projects where it is not possible to formulate the correct response exactly or review it automatically?

You can't create a training like this, because for the response to be counted as correct it must exactly match the control text.

For projects using free text input or attached files, you can make a pre-selection task with non-automatic acceptance. You can admit good performers to your main pool based on their skill.

Can I implement non-automatic acceptance in the training pool?

You can't use non-automatic acceptance in your training pool.

However, you can create a training pool with the Training type based on your main pool and enable non-automatic acceptance there.

Training skill
More than 500 performers passed the training, but the training skill shows only 30.

The pool shows the total number of performers that completed at least one assignment. A training skill can be lost over time if you set repeated training in the pool settings. This setting allows a performer to pass the training again after a certain period if the performer didn't complete any tasks in associated pools or if there was a large time gap between completing tasks (for example, because of the ban). The training skill displays the performers who either recently completed training, or regularly complete your tasks so that the skill doesn't expire.

More performers were trained than the training skill shows

The pool shows the total number of performers that completed at least one assignment. A training skill can be lost over time if you set repeated training in the pool settings. This setting allows a performer to pass the training again after a certain period if the performer didn't complete any tasks in associated pools or if there was a large time gap between completing tasks (for example, because of a ban). The training skill displays the performers who either recently completed training, or regularly complete your tasks so that the skill doesn't expire.

Which parameter affects the skill expiration?

The validity period of the training skills is controlled by the Retry after parameter.

The skill is deleted in the specified number of days if the performer:
  • Has a skill value lower than in the Level required field.
  • Didn't complete any tasks linked to the training during this period.
If their skill expires, your users need to complete the training again.
How do I know when a particular performer got the skill?
  1. Go to the user card.
  2. Click the Profile tab.
  3. Find the required skill in the list and download the history of its changes.
Test and retry
What's the difference between the exam pool that I pay for and the main pool?

An exam pool contains only control tasks. Usually it's small and intended to check how users learned to do your tasks after they read the instructions and completed the training.

Unlike your main pool, you already know the correct responses for every task in this pool. You can set the price to zero. Based on the results of responses to control tasks, you can assign a skill to the users and then specify it in the main pool as a filter. For example, = 80 or = Is missing>. You don't have to create an exam, because the training pool provides enough practice for simple tasks. But many requesters also use exams.

How do I set up a retry pool for my project?

You can create a retry pool similarly to an exam pool. In the pool settings, select the type Retry. In the retry pool filters, specify the upper and lower values of the <main skill> that the users must get in order to be admitted to the retry pool.

For example, if the main pool admits users with a skill of 70 or higher, then you can route the people with a skill between 40 and 69 to the retry pool.

To get a valid “range”, enter the skill twice: with an upper and lower value. For example: <basic skill > <70 and main skill >=40.

We recommend that you don't make your exam and retry pools too lengthy, because performers don't like to do zero-price tasks. 10-20 tasks is enough, depending on complexity.
How do I create an exam with a preset number of correct responses?

To do this, under Test result, go to Recent values to use and specify the number of recent responses from the performer.

Let's say you need to create an exam with three tasks, one task per page. If the performer succeeds in two out of three tasks, they get the skill.

If your task uses assignment review (non-automatic acceptance), to set up such a rule you need to specify 3 for "Total reviewed responses". As you can see in the screenshot, in the first case, all the performers who completed 3 assignments and whose answers are reviewed will get the skill. In the second case, only those who have 2 or 3 assignments accepted will get the skill.

How do I create a training and honey pots with an exam to get an output response other than the control value?

For a control or training assignment to be counted as correct, it must exactly match the control assignment. To do this, you need to normalize the response text using JavaScript: remove spaces, punctuation marks, special characters, and capital letters, and write the result in a separate output field. Now you can match the processed assignment text against your control text.

Another option for selecting performers for a project of this type is assignment review (non-automatic acceptance).