Using exams

An exam is a pool that contains only control tasks. A combination of training tasks, an exam, and the main pool ensures the required level of quality. Assign skills to Tolokers based on the exam results.

Note

If a project has a control pool, Tolokers get access to the general tasks based on the test results rather than training results.

When to use

Use the exam if your tasks are difficult and you want to check how Tolokers cope with them after completing the training.

Features

The number of tasks in the exam must be a multiple of the number of tasks per suite. Otherwise, the Tolokers won't receive the last suite and won't be able to complete the exam.

Example
If the Toloker presses the Skip button when taking the exam, the entire page isn't counted. In this case, the system won't have enough completed tasks to assign a skill to the Toloker.

How to avoid this

Recommendations

Only use tasks with an unambiguous answer to avoid different interpretations.
Make sure that different task response options have about the same frequency in your test.
After the exam, assign the Toloker a skill equal to their percentage of correct responses.
Use different skills for your training and control pools. The system will create the training skill automatically. Create the control skill yourself. Come up with a clear skill name and make the skill public. This way you won't get confused yourself or mislead Tolokers.
Set the same price per task suite for the exam and the main pool. If the price doesn't match, Tolokers won't be able to immediately start completing general tasks after passing the exam.
Don't add too many control tasks. Three to five pages is enough, depending on task complexity.
Don't forget to close the test as soon as you have enough Tolokers to label the main pool, or if labeling of the main pool has ended. Otherwise, the Tolokers will waste their time on the test.

How do I set it up?

Open the project and click Create new pool.
Click Show advanced settings, go to the Additional settings section, and select Exam as the pool type.
Link the training to the exam pool. In the Quality control section, specify the name of the training and the level required. This way, you only let Tolokers take the exam if they passed the training pool.

Example

Note
If the Training list is empty, this probably means you haven't created a training pool in this project yet.
To assign a skill based on the percentage of accepted responses, go to the Quality control section and add a Control tasks rule.

Example
Fill out the Price per task suite, $ box. You can set the price for this type of pool to zero, but it's better to make the exam paid.
Click Create pool.
Add tasks to the pool.

Examples of settings

Main pool

By introducing training and testing, you can ensure that the labeling tasks go to those Tolokers who can deliver high-quality results. But you need to monitor the quality of responses in the main pool as well.

Include some control tasks into your main pool (we recommend that they make up at least 1% of the total number of tasks). Set up quality control so that the Toloker is assigned a skill equal to their percentage of correct responses. That's the main skill of the project.

In the Audience section of the main pool, specify the Toloker requirements:

<exam skill> ≥ 80
<basic skill> ≥ 70 or = Is missing.

In this case, your general tasks are only labeled by top-performing Tolokers or those new to your tasks.

Passing the exam in several attempts

Usually the test is passed once. However, you can give multiple chances to your Tolokers. For example, 10 tasks are enough to pass the test. Upload 5 pages, 10 tasks each, to the exam pool. In the pool settings, add a Control tasks rule.

In the Audience section, specify the Toloker requirement: <exam skill> < 80 or = Is missing.

The Toloker will have 5 attempts to pass the test in this case. If they fail to reach the threshold value of the skill after completing the first 10 tasks, they may retry. To make the second attempt possible (for example, on another day), add the Submitted responses rule in the pool settings.

Retry

If the Toloker doesn't cope with the general tasks after passing the exam, they're banned. However, you can create a retry pool for them. When a Toloker becomes good at retry tasks, you can give them access to the general tasks.

Note

Don't add too many tasks to the retry pool. Otherwise, not all Tolokers will want to complete it.

A retry pool is created similarly to the main pool and consists of control tasks. Go to the project and add a pool of the Retry type. As with the exam, the price for this pool type can be zero.

Use a skill to link the retry pool to the main pool. Use filters to set the main skill values that redirect Tolokers to the retry pool. For example, if the main pool is available to Tolokers with a skill of 70 or higher, send Tolokers to the retry pool if their skill is between 69 and 40.

Based on the Toloker's responses to the retry pool tasks, recalculate their main skill. In the Quality control section, set a Control tasks rule. Leave the Recent control and training task responses to use box empty so that only the retry tasks are used for recalculation.