Toloka documentation

Project 4. Which of the found items is most similar to the original?

In this project, Tolokers compare images of shoes with the original image and decide which pair of images is most similar. To make a comparison in the project, add the results from the second project. To learn how, see Prepare and upload a file with tasks.

Create a project

To create a project, open Toloka for requesters.

In the interface:

  1. Choose a template:

    1. Click Create project.

    2. Select the Image comparison (Side-by-side) template.

  2. Provide general information:

    1. In the Name for Tolokers field, enter Which object is similar to the original one?.

    2. In the Description for Tolokers field, enter Decide which pair of shoes is most similar to the original shoes..

    3. Optionally add a Private comment.

    4. Save the changes.

  3. Edit the task interface in the editor you selected:

    1. Use the ready-made code for this project with pre-configured validation, keyboard shortcuts, and task layout.

      The Toloker won't be able to submit the response until they:

      • Visit the online store.
      • Select one of the response options.

    2. Click Show specifications to see the input and output data fields.

      • Input data fields:

        • image — A link for uploading the image of the original product.
        • left_link — A link to the product in the online store for the left button.
        • right_link — A link to the product in the online store for the right button.

      • Output data field: result — string for saving the Toloker's response.

    1. In the Task Interface, delete the template code from the HTML block and paste the following code:

      <div class="header">
<div class="left caption">
{{button label="Go to site" href=uploaded_link_left size="L"}}
<p class="url">{{uploaded_link_left}}</p>
</div>
<div class="right caption">
<p class="url">{{uploaded_link_right}}</p>
{{button label="Go to site" href=uploaded_link_right  size="L"}}
</div>
</div>

{{img src=image}}

<div class="content clearfix">
<div class="left page">
{{iframe src=uploaded_link_left width="100%" height="700px" real-size=true screenshot=true}}
</div>
<div class="right page">
{{iframe src=uploaded_link_right width="100%" height="700px" real-size=true screenshot=true}}
</div>
</div>

<div class="footer">
{{field type="radio" name="result" label="Left image is better" value="result_left" hotkey="1"}}
{{field type="radio" name="result" label="Right image is better" value="result_right" hotkey="2"}}
</div>

    2. In the js block, edit the code by adding the following lines before OnRender.

      Warning

      Don't remove the existing code.

      getTemplateData: function() {
  var data = TolokaHandlebarsTask.prototype.getTemplateData.apply(this, arguments),
      input = this.getTask().input_values;
  var left_link = input.left_link;
  var right_link = input.right_link;
  var uploaded_link_left = '',
      uploaded_link_right = ''
  if (Math.floor(Math.random() * 2)) {
    uploaded_link_left = left_link
    uploaded_link_right = right_link
  } else {
    uploaded_link_left = right_link
    uploaded_link_right = left_link
  }
  data.uploaded_link_left = uploaded_link_left;
  data.uploaded_link_right = uploaded_link_right;
  data.result_left = uploaded_link_left;
  data.result_right = uploaded_link_right;

  return data;
},

    3. In the css block, insert the following code that is responsible for setting the proportional image size:

      .task {
  display: block;
  text-align:center;
}

.header {
  overflow: hidden;
  background-color: #FFCC00;
}

.caption {
  width: 50%;
}

.url {
  white-space: nowrap;
  overflow: hidden;
  text-overflow: ellipsis;
  max-width: calc(100% - 182px);
  display: inline-block;
  vertical-align: bottom;
}

.button {
  margin: 10px;
  max-width: 182px;
}

.button__label {
  white-space: nowrap;
  overflow: hidden;
  text-overflow: ellipsis;
  max-width: 150px;
}

.content {
  margin: 10px 0;
}

.page {
  display: inline-block;
  width: 50%;
}

.left {
  float: left;
  text-align: left;
}

.right {
  float: right;
  text-align: right;
}

.clearfix {
  overflow: hidden;
  width: 100%;
}

    4. Edit the input and output data in the Data specification block.

      1. Click to switch graphic mode to JSON format.

      2. In the Input data field, enter:

        {
  "image": {
    "type": "url",
    "hidden": false,
    "required": true
  },
  "left_link": {
    "type": "url",
    "hidden": false,
    "required": true
  },
  "right_link": {
    "type": "url",
    "hidden": false,
    "required": true
  }
}

      3. In the Output data field, enter:

        {
  "result": {
    "type": "url",
    "hidden": false,
    "required": true
  }
}

        Learn more about the Specification parameters in Input and output data.

      4. Click to see the Toloker's view of the task.

        Note

        The project preview shows one task with standard data. You can define the number of tasks to show on the page later.

      5. In the window that opens, check if the task options work correctly. In the lower-right corner, click Submit.

      6. Exit preview mode.

        In the lower-left corner, click Exit. If there were errors when testing, check the code blocks that you entered.

  4. Save the changes.

  5. In Instructions for Tolokers, enter the instructions.

    1. Instructions:

      Look at 2 pictures with different shoes and decide which pair of shoes looks most similar to the original pair.
Rely on your own sense of style, but also remember that the shoes look alike if
they are the same color, fabric, length, and style.

    2. Save the changes.

  6. To go back to the Projects page, click Finish editing.

Learn more about working with the project in the Project section.

Create a pool

To create a pool:

  1. Open the page of the project titled Which shoes look more alike.

  2. Click Add a pool on the project page.

  3. Set the Pool name (visible only to you) field. Only you will see this pool name on the project page.

  4. Specify the pool description which will be displayed instead of the project description in the task list for Tolokers. By default, Tolokers see the description from the project settings. To use a different description, uncheck the Use project description box and set Public description. If necessary, click + Private comment to add a private project description that only you will see.

  5. Click Create.

  6. At the Select the audience for your task step, set up filters to select Tolokers for your pool.

    1. Use the Languages filter to select Tolokers with knowledge of the language.

    2. Clear My tasks may contain shocking or pornographic content if your project has none of those.

    3. Tasks in pools will be automatically available in the web version of Toloka and the mobile app. If you want to change the default settings and limit the visibility of the task for any of the versions, add the Client filter and select the desired value: Toloka web version or Toloka for mobile.

    4. Add another filter. Click Add filter.

    5. Find the Skills block in the list and click Choose your skill.

    6. In the Skills field, select Found_shoes.

    7. In the ? field specify =.

    8. Leave the Absent field empty.

  7. (optional) In the Speed/quality balance section, specify the desired quality level. Improving quality may reduce the speed of task completion because the pool will be available to fewer Tolokers. Learn more in the Speed/quality balance section.

  8. At the Set the task price and overlap step, set up how much a single task will cost for you.

    1. In Price per task suite, $, set the amount of money to pay per task suite done by one Toloker. For example, 0.01.

    2. Enter 10 in the Overlap field.

  9. At the Setup quality control step, set quality control rules for more accurate results:

    1. Click Add a quality control rule.

    2. Find the Rules block in the list and choose Fast responses.

    3. Set a rule for fast responses: if the number of fast responses is more than 3, then restrict the Toloker's access to the project for 10 days. Specify Fast responses as the reason for ban.

    Learn more in Quality control.

  10. At the Add optional pool settings step:

    1. At the Add optional pool settings step, specify the Time per task suite, sec.

      The time should be long enough to read the instructions and wait for task data to load. For example, 600 seconds.

    2. Turn on the Keep task order option.

  11. At the Prepare and upload data step, upload your task data.

    1. Download the file received in the third project.

    2. Create pairs for each input image so that you can compare the two found images with the original one and decide which one is more similar.

      You can create pairs manually using a text editor or automatically, like using Python or Jupyter Lab.

      As an example, you can use our results https://tlk.s3.yandex.net/wsdm2020/SbS_Toloka_prep&aggr_data.ipynb.

    3. Open the Which item is more similar? pool.

    4. Click Upload.

    5. Click Drop file here or select, and upload the file you’ve just made.

    6. Click Continue.

    7. Choose Set manually.

    8. In the Tasks per suite field, specify1. You can experiment with the number of tasks per suite.

    9. Click Combine tasks into suites.

  12. At the Double-check your project and try out tasks step, check how the task will look from the Toloker's point of view.

    Note

    This step will be enabled after you complete the previous steps. You can skip this step by clicking Do it later.

  13. Click to start the pool.

Get the results

To get the results:

  1. Click Download results.

    1. In the Status block, only leave the Accepted option enabled.

    2. In the Fields block, leave the link, response ID, and Toloker ID options enabled.

  2. Disable the Separate assignments with empty row option.

  3. Try to use the Bradley-Terry model based on the results. As an example, you can use our https://tlk.s3.yandex.net/wsdm2020/SbS_Toloka_prep&aggr_data.ipynb results.

What's next

In this article: