Survey with Toloka template

In this tutorial, you will learn how to create a survey form and run a survey in Toloka. We will use a project preset designed specifically for this type of data collection.

Survey is a type of data collection task with several types of closed-ended and open-ended questions.

Note

There are also other survey presets in Toloka. Choose the right one with the help of our comparison in the Overview section.

Prerequisites

Before you begin:

Make sure you are registered in Toloka as a requester.
Top up your Toloka account. If you are unsure about the budget, you can do that later in this tutorial. Toloka will display the budget estimate for your project.
See the What tasks can't be placed in Toloka? section. Remember that both Tolokers and requesters are anonymous.
- You should not collect personal data: a full name, contacts, or any other information that can be used to identify a Toloker. To determine whether a Toloker belongs to your target audience, you can ask for their city, date of birth, and age category.
- To communicate with Tolokers, use private messages in Toloka only.

Create a project

We recommend starting with a project preset for easier configuration and better results.

Follow this link, or create a project manually:
1. Click Create a project.
2. Click Do it myself.
3. Select the Survey with Toloka template preset.
Click Choose this preset in the pop-up tab.
In the General information section, add the project name and description:
- Name to show Tolokers: In 2–5 words, state the general idea of the project.
- Description for Tolokers: In a couple of sentences, explain what you expect Tolokers to do. This is just an overview. You will write instructions later.
In the Task interface section, set up what your tasks will look like.

Note
This tutorial uses Template Builder, but you can use the HTML/JS/CSS editor for the same purpose.
1. Copy the code of the example and paste it to the Config section of your project.
2. Edit the code: change questions, hints, and answer options. For example, to configure the text of the answer option, replace the sample answers with your values in the following properties:
  - label: This is the label that Tolokers will see. Make sure it is clear and correct.
  - value: This is the value you will see in the file with the survey results.
  Note
  To learn about other properties of the Config section, their possible values and the impact on the task interface, see the Template Builder Help.
3. The example code contains one question with an obvious answer to check how attentive a Toloker is. You can change this question if needed:
4. Click Data specification → Show specifications. In the following sections, you will see the parameters which match the task interface you set up in Template Builder:
  - Input data: The technical parameter to run a survey.
  - Output data: The parameters for answers given by Tolokers, including the check parameter for the attention check.
In the Instructions for Tolokers editor, enter the instructions Tolokers will see when they start doing your tasks. You can add text, tables, and images to your instructions.

Check the sample text of the instructions, and update it to fit your project.
In the upper-right corner, click Save.

Learn more about working with the project in the Project section.

Create a pool

A pool is a set of tasks sent out to Tolokers at the same time. One project can have many pools. When creating a pool, you set up pricing, audience filters for Tolokers, and quality control.

Click Create new pool on the project page.
Select the value in the Pool type drop-down list.

Pool types

Note
If the price per task suite is zero, you must select the pool type.
Set the Pool name (visible only to you) field. Only you will see this pool name on the project page.
Specify the pool description which will be displayed instead of the project description in the task list for Tolokers. By default, Tolokers see the description from the project settings. To use a different description, uncheck the Use project description box and set Public description. If necessary, click + Private comment to add a private project description that only you will see.
Click Create.
At the Select the audience for your task step, set up filters to select Tolokers for your pool.
1. Clear My tasks may contain shocking or pornographic content if your project has none of those.
2. Select Tolokers who knows the language of your survey:
  
  Also you can select Tolokers based on their location, age, gender, and other parameters. Use the Add filter button and configure the filters you need.
3. Tasks in pools will automatically be available in the web version of Toloka and the mobile app. If you want to change the default settings and limit the visibility of the task for any of the versions, add the Client filter and select the desired value: Toloka web version or Toloka for mobile.
4. Filter out Tolokers who have already taken one survey. It helps prevent the situation when a Toloker completes your survey several times.
  1. Click the Add skill button. Enter the skill name and description, and leave the skill not public.
  2. Click Add filter → My skills and select your skill from the previous step. Configure the filter to recruit Tolokers without the skill.
  You will add a rule that assigns this skill to Tolokers who have already taken one survey later.
At the Setup quality control step, set quality control rules for more accurate results:
1. Click the Review task responses manually toggle, and specify the number of days for checking the task in the Review period in days field (for example, 7).
  
  What is manual review?
2. To filter out Tolokers who complete tasks too fast, click Add a quality control rule → Fast responses. Specify the following values:
  
  These settings mean that a Toloker who completes a task suite in less than 15 seconds will be blocked and won’t be able to access your tasks anymore.
  
  A task suite is a page with a number of tasks. In survey projects, a task suite must contain only one task (one survey).
  
  Tip
  To determine the Minimum time per task suite value, complete your survey and record the time.
3. Add the rule which assigns a skill after one submitted survey. As you’ve set the filter to select Tolokers without this skill in step 3.4, the rule allows you to filter out Tolokers who have already completed your survey.
4. To get the required number of responses, add the Processing rejected and accepted assignments rule. Enter the following values:
  
  This means that if you reject a Toloker’s response during the review, a new survey task will be sent to another Toloker.
5. To automatically accept tasks from those who give correct answers to the attention check question, use the Control tasks rule:
  
  The rule will work if you specify the correct answer to the attention check question in the file with tasks. You will do that later in this tutorial.
At the Add optional pool settings step, specify the Time per task suite, sec.

It should be long enough to read the instructions, wait for the task data to load, and give the answers.
In Price, set up the price for answers from one Toloker.
1. In Price per task suite, set the amount of money to pay for survey completion by one Toloker.
  
  In survey projects, a task suite must contain only one task. Otherwise, a Toloker will see several identical surveys on one page. You will set the number of tasks per suite later in this tutorial.
2. In the Overlap field, define how many respondents you need.
  
  The default overlap (3) means that 3 Tolokers will take your survey. Change this value if needed.
  
  You can also increase overlap when labeling is in progress or when it’s completed.
At the Prepare and upload data step, upload your task data.
1. Create the tasks for Tolokers:
  1. Download the archive with the sample files and unpack it.
  2. Choose a format that is convenient for you and open the file. Change the value of the GOLDEN:check parameter if needed.
  3. Click Drop file here or select, and upload the file from the previous step.
  4. Click Continue.
  5. Tasks are shown to Tolokers in suites. A suite is a single page with multiple tasks.
    
    Go to the Set manually tab. Set one task per suite, that is a Toloker will see only one task with your survey on the page:
2. Click Combine tasks into suites.
At the Double-check your project and try out tasks step, check how the task will look from the Toloker's point of view.

Note
This step will be enabled after you complete the previous steps. You can skip this step by clicking Do it later.

After all the steps, you'll see the Set up is finished and your pool is ready for labeling tip on the pool page.

Start labeling

Make sure you have topped up your account.
Go to the pool page and click the Preview button to check your task. Answer the questions and submit the task.

If the page reloaded successfully, your task works correctly.
To send the tasks to Tolokers and begin the labeling process, click Start labeling.
In the pop-up panel, review the budget and click Launch.

See the results

Track the progress of responses collection on the pool page. You can start the review when the first results are received.

If you’ve set the Control tasks rule in step 4.5 of creating the pool, most of the tasks may be already accepted.

Review the rest of the tasks during the time period specified in the Review period in days field in step 4.1. After that period, all responses are automatically accepted, regardless of whether answers to an attention check are correct or not.

Go to the pool, and click Review assignments.
Choose an assignment.
Check the response to the attention check question, and click Accept or Decline. For rejected responses, enter a comment to specify the reason.

To learn about other ways of review, see the Reviewing Tolokers’ responses section.
After checking all the assignments, сlick the Download results button.
In the Download results dialog, do the following:
1. To download the accepted answers, select Accepted.
2. To make the further processing of the results more convenient, clear Separate assignments with empty row.
3. To download the answers given by not banned Tolokers only, select Exclude assignments by banned Tolokers.
4. Click Download results.

You will get the TSV file. To process the survey results, you can open the file in Excel and save it in the XLSX format.

Helpful tips

You can conduct a survey in two stages. For example, it can be useful when the general filters of the Audience section are not enough to select Tolokers you need. Use skills:
- In the first survey, assign the skill to the relevant Tolokers.
- In the second survey, specify the skill in the Audience section to make the survey available to the relevant Tolokers from the first stage.
You can notify Tolokers about the second survey using messages to those of them who have the necessary skill.
Check the Messages tab regularly to get Tolokers’ reports on problems with your survey.