# Survey on third-party platform

In this tutorial, you will learn how to use Toloka to run a survey created on a third-party platform.

You may need a third-party platform, when:

• Your survey form is too long and complicated to be created in Toloka.

• You are already conducting the survey using other channels to reach out to respondents, and would like to aggregate all the results in one place.

In Toloka, you can place a link to your survey in the task interface. To enable a Toloker to confirm survey completion, show a code to each respondent who takes the survey on the third-party platform. The respondent will need to copy the code and paste it to the task in Toloka. You will see this code in the results file and will be able to check if it is correct.

Note

To learn how to create a survey form in Toloka instead of using a third-party platform, see the Survey with Toloka template tutorial.

## Prerequisites

Before you begin:

• Make sure you are registered in Toloka as a requester.

• Top up your Toloka account. If you are unsure about the budget, you can do that later in this tutorial. Toloka will display the budget estimate for your project.

• See the What tasks can't be placed in Toloka? section. Remember that both Tolokers and requesters are anonymous.

• You should not collect personal data: a full name, contacts, or any other information that can be used to identify a Toloker. To determine whether a Toloker belongs to your target audience, you can ask for their city, date of birth, and age category.

• To communicate with Tolokers, use private messages in Toloka only.

## Choose a preset

We recommend starting with a project preset for easier configuration and better results.

1. In the main menu, choose the Projects tab, and click Create a project.

2. Select the Survey on third-party platform preset.

2. Click Choose this preset in the pop-up tab.

## Create a project

1. Under General information, add the project name and description.

• Name to show Tolokers: In 2–5 words, state the general idea of the project.

• Description for Tolokers: In a couple of sentences, explain what you expect Tolokers to do. This is just an overview. You will write instructions later.

Note

This tutorial uses Template Builder, but you can use the HTML/JS/CSS editor for the same purpose.

1. In the Config section, you can edit the pre-configured code. For example, to change the description, replace the sample text with your value in the label property.

Note

To learn about other properties of the Config section, their possible values and the impact on the task interface, see the Template Builder Help.

2. In the Input data example section, you can see a sample link.

3. Click Data specification → Show specifications. In the following sections, you will see the parameters which match the task interface you set up in Template Builder:

• Output data: Parameters for completion codes entered by Tolokers and for their comments if they have problems with getting a code.

Note

You can also display a survey form in an iframe if a third-party platform allows it. To learn more about the view.iframe component, see the Template Builder Help.

Check the sample text of the instructions, and update it to fit your project.

Note

Do not add a link to your external survey form to the instructions. Specify the link only in the file with tasks. See the section below to learn how you can do that.

4. To save your data and continue, click Create a project.

## Create a pool

A pool is a set of tasks sent out to Tolokers at the same time. One project can have many pools. When creating a pool, you set up pricing, audience filters for Tolokers, and quality control.

You can create a pool using one of the ways:

• From a preset — a new pool with settings for survey:

• Indicate an overlap equal to the required number of Tolokers.

Note

In this case, you need to adjust the overlap, the price of the survey, and the audience settings (including the skill by which the platform will exclude the Tolokers who have already completed the survey once). To learn more about pool settings, see the section below about creating a pool from scratch.

• From scratch — a new pool with default settings.

1. If there are no pools in the project, click Create pool from preset for Survey on third-party platform on the Pools tab. Otherwise, click Add a pool → From Survey on third-party platform preset at the right side of the project page.

2. Under General information, set the Pool name.

3. Under Audience, check the pre-configured filters and correct them if necessary.

4. Clear My tasks may contain shocking or pornographic content if your project has none of those.

5. Under Quality control, check the quality control rules and correct them if necessary.

7. In Price, set up the price for answers from one Toloker.

1. In Price per task suite, set the amount of money to pay for survey completion by one Toloker.

In survey projects, a task suite must contain only one task. Otherwise, a Toloker will see several identical surveys on one page. You will set the number of tasks per suite later in this tutorial.

2. In the Overlap field, define how many respondents for each task you need.

• If you are going to use one static link to your survey, the overlap should be equal to the number of respondents you need for the survey as a whole. You can also increase overlap when labeling is in progress or when it’s completed.

• If you are going to provide many survey links and each of them will be given to one Toloker only, the overlap should be set to 1.

You will prepare the file with one or several survey links in the next step.

8. To save the settings and continue, click Create pool.

1. If there are no pools in the project, click Create new pool from scratch on the Pools tab. Otherwise, click Add a pool → From scratch at the right side of the project page.

2. Under General information, set the Pool name.

3. Under Audience, set up filters to select Tolokers for your pool.

1. Clear My tasks may contain shocking or pornographic content if your project has none of those.

2. Select Tolokers who knows the language of your survey:

Additionally, you can select Tolokers based on their location, age, gender, and other parameters. Use the Add filter button and configure the filters you need.

3. Tasks in pools will automatically be available in the web version of Toloka and the mobile app. If you want to change the default settings and limit the visibility of the task for any of the versions, add the Client filter and select the desired value: Toloka web version or Toloka for mobile.

4. Filter out Tolokers who have already taken one survey. It helps prevent the situation when a Toloker completes your survey several times.

1. Click the Add skill button. Enter the skill name and description, and leave the skill not public.

2. Click Add filter → My skills and select your skill from the previous step. Configure the filter to recruit Tolokers without the skill.

You will add a rule that assigns this skill to Tolokers who have already taken one survey later.

4. Under Quality control, set quality control rules for more accurate results.

1. Click the Non-automatic acceptance toggle, and specify the number of days for checking the task in the Review period in days field (for example, 7).

What is non-automatic acceptance (assignment review)?

The non-automatic acceptance option allows you to review completed assignments before accepting them and paying for them. If the Toloker didn't follow the instructions, you can reject the assignment.

2. To filter out Tolokers who complete tasks too fast, edit the pre-configured Fast responses rule. Specify the following values:

These settings mean that a Toloker who completes a task suite in less than 15 seconds will be blocked and won’t be able to access your tasks anymore.

A task suite is a page with a number of tasks. In survey projects, a task suite must contain only one task (one survey).

Tip

To determine the Minimum time per task suite value, complete your survey and record the time.

3. Delete the pre-configured Majority vote rule.

4. To get the required number of responses, add the Processing rejected and accepted assignments rule. Enter the following values:

This means that if you reject a Toloker’s response during the review, a new survey task will be sent to another Toloker.

5. Add the rule which assigns a skill after one submitted survey. As you’ve set the filter to select Tolokers without this skill in step 3.4, the rule allows you to filter out Tolokers who have already completed your survey.

6. To automatically accept tasks from those who entered a completion code correctly, use the Control tasks rule:

The rule will work if you specify the correct completion code in the file with tasks. You will do that later in this tutorial.

Warning

Time per task suite should be long enough to read the instructions, wait for the task data to load, give the answers on the third-party website, and copy and paste the completion code. Otherwise, Tolokers won’t be able to submit your tasks.

6. In Price, set up the price for answers from one Toloker.

1. In Price per task suite, set the amount of money to pay for survey completion by one Toloker.

In survey projects, a task suite must contain only one task. Otherwise, a Toloker will see several identical surveys on one page. You will set the number of tasks per suite later in this tutorial.

2. In the Overlap field, define how many respondents for each task you need.

• If you are going to use one static link to your survey, the overlap should be equal to the number of respondents you need for the survey as a whole. You can also increase overlap when labeling is in progress or when it’s completed.

• If you are going to provide many survey links and each of them will be given to one Toloker only, the overlap should be set to 1.

You will prepare the file with one or several survey links in the next step.

7. To save the settings and continue, click Create pool.

2. Create the tasks for Tolokers:

• Template in XLSX
• Template in TSV
• Template in JSON
2. Open the downloaded file. It must have one parameter which name equals INPUT:survey_link, and the values are links.

In this case, the overlap you’ve set in step 6.2 of creating the pool should be equal to the number of respondents you need.

• Provide many survey links to give each of them to one Toloker.

In this case, the number of links in the file should be equal to the number of respondents you need. The overlap should be set to 1.

You can use different survey links if it’s necessary to generate different codes for each Toloker. It’s also possible to specify the same link many times.

Note

• Different completion codes for each Toloker help minimize the risk of fraud.

• If you provide many survey links, make sure that you’ve configured the Submitted responses rule and the Time per task suite, sec value is enough (see steps 4.5 and 5 of creating the pool).

For example, the XLSX file should look as follows:

INPUT:url
https://survey.com/mysurvey

INPUT:url
https://survey.com/mysurvey/OIN7YV2TP
https://survey.com/mysurvey/YUT9IJ3PL
https://survey.com/mysurvey/RDV3TG8OJ

2. If you know the correct completion codes beforehand, add the GOLDEN:code parameter to the file and specify its values:

INPUT:url GOLDEN:code
https://survey.com/mysurvey 123456

INPUT:url GOLDEN:code
https://survey.com/mysurvey/OIN7YV2TP 123456
https://survey.com/mysurvey/YUT9IJ3PL 234567
https://survey.com/mysurvey/RDV3TG8OJ 345678


In this case, you can automatically accept tasks from those who entered a completion code correctly. Use the Control tasks rule from step 4.6 of creating the pool.

3. Click Drop file here or select, and upload the file you’ve just made.

4. Click Continue.

5. Tasks are shown to Tolokers in suites. A suite is a single page with multiple tasks.

Go to the Set manually tab. Set one task per suite, that is a Toloker will see only one task with your survey on the page:

6. Click Combine tasks into suites.

## Start labeling

1. Make sure you have topped up your account.

2. Go to the pool page and click the Preview button to check your task.

2. Enter the code and submit the task.

3. To send the tasks to Tolokers and begin the labeling process, click Start labeling.

4. In the pop-up panel, review the budget and click Launch.

## See the results

Track the progress of responses collection on the pool page. You can start the review when the first results are received.

If you’ve set the Control tasks rule in step 4.6 of creating the pool, most of the tasks may be already accepted.

Review the rest of the tasks during the time period specified in the Review period in days field in step 4.1. After that period, all responses are automatically accepted, regardless of whether completion codes are correct or not.

1. Go to the pool, and click Review assignments.

2. Choose an assignment.

3. Check the completion code, and click Accept or Decline. For rejected responses, enter a comment to specify the reason.

To learn about other ways of review, see the Reviewing Tolokers’ responses section.

Note

• Do not rush to reject responses that have not passed automatic completion via codes. Some codes will be incorrect indeed, but often Tolokers add extra spaces accidentally or leave comments in a text field which is intended for a code.

• If there were technical problems (for example, a Toloker had filled an external survey form, but didn’t have time to submit a task in Toloka), you still can pay the Toloker. To do this, use the rewarding system.

• You can use questions with known answers to check that Tolokers pay attention. In that case, even if a Toloker answers incorrectly, it’s better to still provide them a special completion code, different from the correct one. When you see that special code in the results, you can reject the assignment. The Toloker won’t be dissatisfied that you didn't allow them to finish the task.

• You can conduct a survey in two stages. For example, it can be useful when the general filters of the Audience section are not enough to select Tolokers you need. Use skills:

• In the first survey, assign the skill to the relevant Tolokers.

• In the second survey, specify the skill in the Audience section to make the survey available to the relevant Tolokers from the first stage.

You can notify Tolokers about the second survey using messages to those of them who have the necessary skill.

• Sometimes a survey may follow different scenarios depending on answers given to the previous questions. In this case, make sure that all scenarios have a completion code in the end.

• Check the Messages tab regularly to get Tolokers’ reports on problems with your survey.