Toloka documentation

Project 3. Does the item found look similar to the original?

In this project, Tolokers will compare shoes from the original image (shown in project 1) with shoes found in project 2.

Warning

Since the Assignment review option is enabled in the pool settings for project 2, you should complete all the steps and check the Tolokers' responses within the time limit set in the Review period field.

Create a project

To create a project, open Toloka for requesters.

In the interface:

  1. Choose a template:

    1. Click Create project.

    2. Select the Image classification template.

  2. Provide general information:

    1. In the Name for Tolokers field, enter Are these shoes similar to each other?.

    2. In the Description for Tolokers field, enter Take a look at two pairs of shoes and decide whether they look similar or not..

    3. Optionally add a Private comment.

    4. Save the changes.

  3. Edit the task interface in the editor you selected:

    1. Use the ready-made code for this project with pre-configured validation and task layout.

      The Toloker won't be able to submit the response until they:

      • Click the button to go to the online store.
      • Select one of the response options.
    2. Click Show specifications to see the input and output data fields.

      • Input data fields:

        • image — A link for uploading the image of the original product.
        • found_link — A link to the product in the online store.
        • assignment_id — A link to previous projects.
      • Output data field: result — string for saving the Toloker's response.

    1. In the Task Interface, delete the template code from the HTML block and paste the following code:

      {{img src=image height="400px"}}
      {{iframe src= found_link height="600px"}}
      
      <p>Check that the uploaded image matches the product in the store.</p>
      {{button label="Check the product"  href=found_link action=true}}
      
      <p>Are <b>these shoes</b> similar to each other?</p>
      <p>The shoes must be similar in color, material, length, and style.</p>
      {{field type="radio" name="result" value="Yes" label="Yes"}}
      {{field type="radio" name="result" value="No" label="No"}}
      
    2. Leave the JS block unchanged.

    3. In the css block, insert the following code that is responsible for setting the proportional image size:

      .task {
        display: block;
      }
      .img {
        float: left;
        width: 50%;
      }
      .iframe {
        float: left;
        width: 50%;
      }
      
    4. Edit the input and output data in the Data specification block.

      1. Click to switch graphic mode to JSON format.

      2. In the Input data field, enter:

        {
          "image": {
            "type": "url",
            "hidden": false,
            "required": true
          },
          "found_link": {
            "type": "url",
            "hidden": false,
            "required": true
          },
          "assignment_id": {
            "type": "string",
            "hidden": true,
            "required": true
          }
        }
        
      3. In the Output data field, enter:

        {
          "result": {
            "type": "string",
            "hidden": false,
            "required": true
          }
        }
        

        Learn more about the Specification parameters in Input and output data.

      4. Click to see the Toloker's view of the task.

        Note

        The project preview shows one task with standard data. You can define the number of tasks to show on the page later.

      5. In the window that opens, check if the task options work correctly. In the lower-right corner, click Submit.

      6. Exit preview mode.

        In the lower-left corner, click Exit. If there were errors when testing, check the code blocks that you entered.

  4. Save the changes.

  5. In Instructions for Tolokers, enter the instructions.

    1. Instructions:

      Take a look at the pictures that show two pairs of shoes. Decide whether they look similar or not.
      
      Shoes are considered similar if they have a similar color, fabric, length, and style.
      If you don't see a pair of shoes in any of the pictures, click **Yes**.
      
    2. Save the changes.

  6. To go back to the Projects page, click Finish editing.

Learn more about working with the project in the Project section.

Create a pool

To create a pool:

  1. Open the page of the project titled Are these shoes similar to each other?.

  2. Click Add a pool on the project page.

  3. Set the Pool name (visible only to you) field. Only you will see this pool name on the project page.

  4. Specify the pool description which will be displayed instead of the project description in the task list for Tolokers. By default, Tolokers see the description from the project settings. To use a different description, uncheck the Use project description box and set Public description. If necessary, click + Private comment to add a private project description that only you will see.

  5. Click Create.

  6. At the Select the audience for your task step, set up filters to select Tolokers for your pool.

    1. Use the Languages filter to select Tolokers with knowledge of the language.

    2. Clear My tasks may contain shocking or pornographic content if your project has none of those.

    3. Tasks in pools will be automatically available in the web version of Toloka and the mobile app. If you want to change the default settings and limit the visibility of the task for any of the versions, add the Client filter and select the desired value: Toloka web version or Toloka for mobile.

    4. Add another filter. Click Add filter.

    5. Find the Skills block in the list and click Choose your skill.

    6. In the Skill field, select Found_shoes.

    7. In the ? field specify =.

    8. Leave the Absent field empty.

  7. (optional) In the Speed/quality balance section, specify the desired quality level. Improving quality may reduce the speed of task completion because the pool will be available to fewer Tolokers. Learn more in the Speed/quality balance section.

  8. At the Set the task price and overlap step, set up how much a single task will cost for you.

    1. In Price per task suite, $, set the amount of money to pay per task suite done by one Toloker. For example, 0.01.

    2. Enter 3 in the Overlap field.

  9. At the Setup quality control step, set quality control rules for more accurate results:

    1. Click Add a quality control rule.

    2. Find the Rules block in the list and choose Control tasks.

    3. Set a rule for control tasks: If the number of responses to control questions is greater than or equal to 3 and the percentage of correct responses to control questions is less than 60, then restrict the Toloker's access to the project for 10 days. Specify the Control task as a reason.

    4. (optional) Add another quality control rule. Click Add a quality control rule.

    5. Find the Rules block in the list and choose Fast responses.

    6. In the Minimum time per task suite field, specify 60.

    7. Set a rule for fast responses: if the number of fast responses is more than 1, then restrict the Toloker's access to the project for 10 days. Specify Fast responses as the reason.

    Learn more in Quality control.

  10. At the Add optional pool settings step:

    1. At the Add optional pool settings step, specify the Time per task suite, sec.

      The time should be long enough to read the instructions and wait for task data to load. For example, 300 seconds.

    2. Turn on the Keep task order option.

  11. At the Prepare and upload data step, upload your task data.

    1. Wait for the pool from project 2 to complete.

    2. Open the pool page from project 2.

    3. Click Download results.

      1. In the Status block, only leave the Not checked option enabled.

      2. In the Fields block, only leave the response ID option enabled.

      3. Disable the Separate assignments with empty row option.

      4. Click Download results.

    4. Prepare a file with tasks.

      1. Leave the INPUT:image column unchanged.

      2. Rename the OUTPUT:found_link column to INPUT:found_link.

      3. Rename the ASSIGNMENT:assignment_id column to INPUT:assignment_id.

      4. Delete all the other columns.

      5. Save the file.

    5. Upload the resulting file to the Are these shoes similar to each other? pool.

      1. Open the Are these shoes similar to each other? pool.

      2. Click Upload.

      3. Click Drop file here or select, and upload the file you’ve just made.

      4. Choose Smart mixing.

      5. In the General tasks field, specify 9.

      6. In the Training tasks field, specify 0.

      7. In the Control tasks field, specify 1.

      8. Click Upload.

      9. Click Combine tasks into suites.

  12. Create control tasks at the Add control tasks for checking performance step. To do it, add correct answers to some of your tasks.

    1. In the left column, turn on the result option.

    2. Choose the correct answer to the question.

    3. Click Save and go to next.

    4. Click Are these shoes similar to each other? to exit task markup mode.

      Note

      In small pools, control tasks should be 5–10% of all tasks. Include different versions of correct responses in equal amounts. See the distribution of responses on the Edit tasks page, Control tasks tab.

  13. At the Double-check your project and try out tasks step, check how the task will look from the Toloker's point of view.

    Note

    This step will be enabled after you complete the previous steps. You can skip this step by clicking Do it later.

After all the steps, you'll see the Set up is finished and your pool is ready for labeling tip on the pool page.

Download the reviewed results

To get the results:

  1. Next to the Download results button, click .

  2. Choose the Dawid-Skene aggregation model. Learn more about Result aggregation based on the Dawid-Skene model.

  3. In the window that opens, click Yes.

  4. At the top of the page, click View operations list.

    Note

    To track the progress, refresh the page from time to time. Aggregation takes from 5 to 20 minutes. You can start designing another project meanwhile.

  5. When the operation is complete, download the file with the results. To do this, click Download in the Files column.

  6. Use the file with the results from project 2.

Check the completed tasks

You can check the results in two ways:

  • In the file with the results.
  • In the pool interface.

Review assignments in the file with the results

To review the submitted assignments:

  1. In the text or spreadsheet editor, open the file that you received after aggregating the results.

  2. Prepare the file:

    1. Add a column named ACCEPT:verdict with the review results.

    2. Add a column named ACCEPT:comment with comments for Tolokers if responses were rejected. For example, comment on which part of the instructions wasn't followed.

    3. Rename the INPUT:assignment_id column to ASSIGNMENT:assignment_id.

  3. Fill in the ACCEPT:verdict: and ACCEPT:comment: columns:

    • If the aggregate result of the assignment is OK, put + to accept it.

    • If the aggregate result of the assignment is incorrect or it doesn't open, put - to reject it. Specify the reason for assignment rejection in the ACCEPT:comment: field (for example, The item provided is incorrect or doesn't open.).

  4. Delete all the other columns.

  5. Save the file.

  6. Open the pool page in project 2.

  7. Click Review assignments.

  8. Click Upload results.

  9. In the window that opens, choose the file with the results to upload and click Open.

  10. In the window that opens, compare the number of assignments in the Processed successfully field with that in the Total submitted field on the pool page.

  11. Click the Add button.

  12. In the window that opens, click Close.

  13. When setting up the pool in project 2, you enabled the Recompletion of the rejected tasks option.

    In this case, the pool automatically reopens and the assignments are reassigned to other Tolokers. When they're completed, send the results for verification. Then download the results, check them, and upload the reviewed results. You can reject assignments as many times as you want to get more accurate results.

Review assignments in the pool interface

To check submitted assignments:

  1. Open the pool page in project 2.

  2. Click View assignments.

  3. Hover over the line of the assignment you want to check.

  4. In the Status column, you'll see the buttons for accepting () or rejecting () the completed assignment. If you reject the assignment, enter a comment in the window that opens and click Done.

What's next

  • Create Project 4 to compare matching images.