Surveys

Tip.

Run the project in the Sandbox first. This helps you avoid making mistakes and spending money on a task that isn't working right.

  1. Create a project
  2. Add a task pool
  3. Upload tasks
  4. Set up quality control
  5. Start the pool and get the results
  6. Troubleshooting

Surveys are designed to collect information for research purposes. The number of respondents is set in overlap, and the requirements for them are set with filters.

You may need additional projects for your task, such as dataset pre-check or checking performers' responses. Learn more about this in Decomposition of the task.

Suppose you want to know where Russian-speaking respondents have breakfast and what they prefer. To do this, create a single task on a page and in the pool, so the performer sees several questions with response options.

Example of a prepared task

To run tasks and get responses:

Create a project

The project defines what the task will look like for a performer.

  1. Choose a template:

    1. Click Create project.

    2. Select the Survey template.

  2. Provide general information:

    1. Enter a clear name and a short description for the project. Performers will see this in the task list.

    2. Optionally add a Private comment.

    3. Click Save.

  3. Edit the task interface in the editor you selected:

    Template Builder
    HTML/CSS/JS editor
    1. The task interface describes how the elements should be arranged in the task.

      Use the ready-made code for this project with pre-configured validation and task layout. The performer won't be able to submit the response until they answer all the questions.

      Learn more about setting up conditions in Template Builder.

    2. Click Show specifications to see the input and output data fields.

      Input data fields are created from the code on the Example of input data tab.

      The output data fields depend on the components that use data.output and values supported by it.

      Learn more about input and output data fields in the Template Builder Help.

      • Input data field — The theme string to upload a survey. It is not used in the interface and can contain any text.
      • Output data fields:
        • The required place string to record the performer's response to the question “Where do you usually have breakfast?”.;
        • The required string food to record the performer's response to the question “What do you eat for breakfast most often?”.;
        • The required string drink to record the performer's response to the question “What do you drink for breakfast (besides water)?”.
    3. Save the changes.

    1. Define which objects you are going to pass to the performers and receive from them in response. To do this, add input and output fields in the Data specification section.

      What are input and output data?

      Input data is types of objects that are passed to the performer for completing the task. For example, this could be a text, an image, or geographic coordinates.

      Output data is types of objects that you receive after the task is completed. For example, this could be one of several response options, typed text, or an uploaded file.

      Learn more about input and output data fields.

      Delete all the output data fields and add the fields you need:

      • Input data field — The theme string to upload a survey. It is not used in the interface and can contain any text.
      • Output data fields:
        • The required place string to record the performer's response to the question “Where do you usually have breakfast?”.;
        • The required string food to record the performer's response to the question “What do you eat for breakfast most often?”.;
        • The required string drink to record the performer's response to the question “What do you drink for breakfast (besides water)?”.
    2. Create the task interface in the HTML block. It describes how the task elements should be arranged in the task.

      You can use standard HTML tags and special expressions in double curly brackets for input and output data fields.

      <div>  <div>Where do you usually have breakfast?</div>
        {{field type="radio" name="place" value="home" label="At home"}}
        {{field type="radio" name="place" value="university" label="At school"}}
        {{field type="radio" name="place" value="work" label="At work"}}
        {{field type="radio" name="place" value="cafe" label="In a cafe"}}<br/><br/><br/>
      </div>
      <div>
        <div>What do you usually eat for breakfast?</div>
        {{#field type="select" name="food" placeholder="" width="300px"}}
          {{select_item value="kasha" text="Porridge/Muesli/Cold cereal"}}
          {{select_item value="egg" text="Eggs (fried/boiled/omelet)"}}
          {{select_item value="sandwich" text="Sandwich/Pastry/Crepes/Pancakes"}}
          {{select_item value="curd" text="Cottage cheese/Yogurt/Other dairy"}}
          {{select_item value="other" text="Other"}}
        {{/field}}<br/><br/><br/>
      </div>
      <div>
        <div>What do you usually drink at breakfast?</div>
        {{field class="option" type="radio" name="drink" value="coffee" label="Coffee/Tea"}}
        {{field class="option" type="radio" name="drink" value="milk" label="Milk"}}
        {{field class="option" type="radio" name="drink" value="juice" label="Juice"}}
        {{field class="option" type="radio" name="drink" value="other" label="Other"}}<br/><br/><br/>
      </div>
      <div>
        Thank you for completing our survey!
      </div>
      Copied to clipboard

      This means that the performer will see three questions with response options.

      Clear the JavaScript and CSS blocks.

    3. Click to see the performer's view of the task.

      Note. The project preview shows one task with standard data. You can define the number of tasks to show on the page later.
    4. Save the changes.

  4. For a survey, instructions aren't required. Place the recommendations for filling out the survey in the task interface.
    Note. Don't leave your contact information. This contradicts the Requester Agreement.
  5. To go back to the Projectspage, click Finish.

Learn more about working with the project in Project.

Add a task pool

A pool is a set of paid tasks sent out for completion at the same time.

  1. Open the project and click Add pool.
  2. Give the pool any convenient name. It is available only to you — the performer will only see the name of the project.
  3. Set the price per task suite (for instance, $0.01). In a survey, a task suite must contain one task. If you need help rating tasks, read about setting up pricing and see examples of prices for different types of tasks.
  4. Specify the Time for task completion (for instance, 600 seconds).
  5. Set the Overlap, which is the number of performers to complete the same task. At first, set a small number like 100. If you see that this is not enough for sampling, increase the overlap.
  6. Add Filters to choose performers.
    Run the task both in the web version and in mobile apps to speed up task completion. To do this, add the Client filter.
  7. Save the pool.

Upload tasks

Download the sample upload file. You can find it on the pool page. At the top-left of the page, there are links to TSV files with regular, control, and training tasks. Use it to prepare your own file with tasks.

  1. Click Upload. In the window that opens, you can also download a sample TSV file by clicking Sample file for uploading tasks.
    What is TSV?
    A TSV file presents a table as a text file in which columns are separated by tabs.

    You can work with it both in a table editor and a text editor, and then save it to the desired format. More about working with a TSV file. There is a CSV format that is similar to TSV, but you should use a TSV file for uploading.

  2. Add input data in it. The header of the input data column contains the word INPUT. To upload a task, add any text to the INPUT column. The string from the input data field isn't used in the interface. Leave the other columns empty.
  3. Load tasks by choosing Set manually and setting 1 task per page.

Set up quality control

Quality control rules allow you to filter out inattentive performers. You can configure quality control both in the project and in the pool.

    Go to pool editing (the Edit button in the upper-right corner of the page) and click Add Quality Control Rule.

    You can copy quality control settings from another pool. To do this, click Copy settings from in the Users filter section.

  1. Add the Fast responses rule and enter the following values:

    This means that a user who completes a task suite in less than 10 seconds will be banned for 10 days and won't be able to complete project tasks.

  2. Add the Banned users' reassessment block.

    For surveys, always add both rules, since they're only effective when paired up for such tasks. Surveys usually only have one task suite. This rule complements fast responses so that you get the necessary number of responses, and not just ban those who respond too quickly.

    Attention. Recompletion of assignments from banned users increases the overlap, hence the budget, too.

    Reject assignments according to the conditions described in the instructions. To maintain the reputation of a good requester, don't automatically reject fast responses like that. It's easy to fill out a simple surveys in less than 20 seconds.

    This means that if a performer is banned for fast responses and the responses aren't taken into account, this task is reassigned to another user for completion.

Start the pool and get the results

  1. Start the pool by clicking .
  2. Track the completion of tasks in the Pool statistics section.
  3. When the pool is completed, launch aggregation of results. To do this, find the Download results button and click  → Dawid-Skene aggregation model next to it.

    Aggregation of responses is necessary to get a complete picture of all results. Learn more about aggregation.

  4. Track the aggregation progress on the Operations page. When the process is completed, click Download.
  5. If the performers' responses aren't enough for analysis, increase the Overlap to receive more responses.
    How do I increase overlap?
    1. Stop the pool by clicking .
    2. In the upper-right corner, click Edit.
    3. Change the value in the Overlap block.
    4. Click the Save button and start the pool.

Troubleshooting

How do I create a simple survey with no options, where the user answers an open-ended question?
  1. Create a project from an empty template.
  2. Write your question in the HTML block.
  3. If you need an extensive response, add the required number of text entry components.
  4. If you need a short response, add the required number of string entry components.
  5. Come up with a name for each of the components and create a matching number of string-type output fields with the same names. They will be used to save responses.
  6. Make all the output fields mandatory.
  7. Make the input field auxiliary. It will only be used to create a TSV file with tasks.
  8. Come up with a name for the input field and set the string type for it (see the step-by-step guide on creating a survey).
How do I create a survey with a single question based on the preset survey template?

The survey includes an auxiliary input field. You can use it to pass any information, and it won't be visible to performers.

Upload one task and use the overlap parameter to enter the number of people you want to survey.