Using exams

An exam is a pool that contains only control tasks. A combination of training tasks, an exam, and the main pool ensures the required level of quality. Assign skills to Tolokers based on the exam results.

Note. If a project has a control pool, Tolokers get access to the general tasks based on the test results rather than training results.

When to use

Use the exam if your tasks are difficult and you want to check how Tolokers cope with them after completing the training.

Features

  • The number of tasks in the exam must be a multiple of the number of tasks per suite. Otherwise, the Tolokers won't receive the last suite and won't be able to complete the exam.
    Example
    If the exam consisted of two suites, 10 tasks each, and you increased the number of tasks per suite to 12, there won't be enough tasks for two full suites. Tolokers won't receive the second suite and won't be able to complete the exam.
  • If the Toloker presses the Skip button when taking the exam, the entire page isn't counted. In this case, the system won't have enough completed tasks to assign a skill to the Toloker.
    How to avoid this
    • In project editing mode, click Show common interface elements and disable the Skip button.
    • Add more tasks to the pool than you need to calculate the skill, specify the required number of responses, and set the following Toloker requirement: <exam skill> = Is missing.
    • Place all your tasks on one page.

Recommendations

  • Only use tasks with an unambiguous answer to avoid different interpretations.
  • Make sure that different response options have about the same frequency in your test.
  • After the exam, assign the Toloker a skill equal to their percentage of correct responses.
  • Use different skills for your training and control pools. The system will create the training skill automatically. Create the control skill yourself. Come up with a clear skill name and make the skill public. This way you won't get confused yourself or mislead Tolokers.
  • Set the same price per task suite for the exam and the main pool. If the price doesn't match, Tolokers won't be able to immediately start completing general tasks after passing the exam.
  • Don't add too many control tasks. Three to five pages is enough, depending on task complexity.
  • Don't forget to close the test as soon as you have enough Tolokers to label the main pool, or if labeling of the main pool has ended. Otherwise, the Tolokers will waste their time on the test.

How do I set it up?

  1. Open the project and click Add a pool.
  2. Click Show advanced settings, go to the Additional settings section, and select Exam as the pool type.
  3. Link the training to the exam pool. In the Quality control section, specify the name of the training and the level required. This way, you only let Tolokers take the exam if they passed the training pool.
    Example

    Tolokers can take the exam if they answered at least 80% of the training questions correctly.

    Note. If the Training list is empty, this probably means you haven't created a training pool in this project yet.
  4. To assign a skill based on the percentage of accepted responses, go to the Quality control section and add a Control tasks rule.
    Example

    For example, completing 10 tasks is enough to pass the exam. The Toloker will be assigned an exam skill value equal to their percentage of correct responses.

  5. Fill out the Price per task suite, $ box. You can set the price for this type of pool to zero, but it's better to make the exam paid.
  6. Click Create pool.
  7. Add tasks to the pool.

Examples of settings

Main pool

By introducing training and testing, you can ensure that the labeling tasks go to those Tolokers who can deliver high-quality results. But you need to monitor the quality of responses in the main pool as well.

Include some control tasks into your main pool (we recommend that they make up at least 1% of the total number of tasks). Set up quality control so that the Toloker is assigned a skill equal to their percentage of correct responses. That's the main skill of the project.

In the Audience section of the main pool, specify the Toloker requirements:
  • <exam skill> ≥ 80
  • <basic skill> ≥ 70 or = Is missing.
In this case, your general tasks are only labeled by top-performing Tolokers or those new to your tasks.
Passing the exam in several attempts

Usually the test is passed once. However, you can give multiple chances to your Tolokers. For example, 10 tasks are enough to pass the test. Upload 5 pages, 10 tasks each, to the exam pool. In the pool settings, add a Control tasks rule.

In the Audience section, specify the Toloker requirement: <exam skill> < 80 or = Is missing.

The Toloker will have 5 attempts to pass the test in this case. If they fail to reach the threshold value of the skill after completing the first 10 tasks, they may retry. To make the second attempt possible (for example, on another day), add the Submitted responses rule in the pool settings.

Retry

If the Toloker doesn't cope with the general tasks after passing the exam, they're banned. However, you can create a retry pool for them. When a Toloker becomes good at retry tasks, you can give them access to the general tasks.

Tip. Don't add too many tasks to the retry pool. Otherwise, not all Tolokers will want to complete it.

A retry pool is created similarly to the main pool and consists of control tasks. Go to the project and add a pool of the Retry type. As with the exam, the price for this pool type can be zero.

Use a skill to link the retry pool to the main pool. Use filters to set the main skill values that redirect Tolokers to the retry pool. For example, if the main pool is available to Tolokers with a skill of 70 or higher, send Tolokers to the retry pool if their skill is between 69 and 40.

Based on the Toloker's responses to the retry pool tasks, recalculate their main skill. In the Quality control section, set a Control tasks rule. Leave the Recent control and training task responses to use box empty so that only the retry tasks are used for recalculation.

A Toloker who gives enough correct responses gets access to the main pool again.

Troubleshooting

Get initial consultation
What's the difference between the exam pool that I pay for and the main pool?

An exam pool contains only control tasks. Usually it's small, and intended to check how Tolokers have learned to do your tasks after they read the instructions and have completed the training.

Unlike your main pool, you already know the correct responses for every task in this pool. You can set the price to zero. Based on the results of the control tasks, you can assign a skill to the Tolokers and then use it in the main pool as a filter. For example, ≥ 80 or ≠ Is missing. You don't have to create an exam, because the training pool provides enough practice for simple tasks. But many requesters also use exams.

How do I set up an exam so that different people can take it without running out of tasks?

When you load tasks, use smart mixing. In this case, you'll have infinite overlap in your exam.

However, this poses the risk that you might spend a lot of money on the exam. You might want to open this pool only when the main pool opens, and close it when labeling of the main pool ends.

Request for training in crowdsourcing and working with Toloka