Project 3. Are the bounding boxes correct?

In this customizable project, Tolokers will determine if traffic signs were selected correctly on the images in project 1, and submit the responses in project 2.

  1. Create a project
  2. Create a pool
  3. Prepare and upload a TSV file with the results
  4. Download the reviewed results
  5. Check the completed tasks

Create a project

In the interface:
  1. Choose a template:

    1. Click + Create project.

    2. Select the Object recognition & detection template.
  2. Provide general information:

    1. In the Name for Tolokers field, enter Are the traffic signs outlined correctly?.
    2. In the Description for Tolokers field, enter Look at the image and decide whether the traffic signs are outlined correctly or not.
    3. Optionally add a Private comment.
    4. Click Save.
  3. Edit the task interface:

    1. Prepare Task interface:

      1. Connect the $TOLOKA_ASSETS/js/image-annotation.js library (click in the Task interface block on the project page).

      2. In the html block, replace the current code with the following:

        <!-- editor for selecting objects that lets you add an area in advance -->
        {{field type="image-annotation" name="object" src=image annotations=selection}}
        
        <!-- buttons for responses -->
        {{field type="radio" name="result" value="OK" label="Correct" hotkey="1"}}
        {{field type="radio" name="result" value="BAD" label="Incorrect" hotkey="2"}}
        Copied to clipboard
      3. In the css block, replace the code with the following:

        /* hide the button for polygon selection */
        .image-annotation-editor__shape-polygon {
          display: none;
        }
        
        /* adjust the interface height */
        .image-annotation-editor__annotation-layer {
          height: max-content;
        }
        Copied to clipboard
    2. Configure the Data specification section:

      1. Click to switch graphic mode to JSON format.

      2. Remove the template code from the Input data field and enter the following code:

        {
          "image": {
            "type": "url",
            "hidden": false,
            "required": true
          },
          "selection": {
            "type": "json",
            "hidden": false,
            "required": false
          },
          "assignment_id": {
            "type": "string",
            "hidden": true,
            "required": true
          }
        }
        Copied to clipboard
      3. Remove the template code from the Output data field and enter the following code:

        {
          "result": {
            "type": "string",
            "hidden": false,
            "required": true
          }
        }
        Copied to clipboard

      Learn more about Specifications in Input and output data.

  4. To test the interface:

    1. Click to see the Toloker's view of the task.

      Note. The project preview shows one task with standard data. You can define the number of tasks to show on the page later.
    2. Click Change input data.
    3. In the selection field, add an example of input data:
      [{"data":{"p1":{"x":0.472,"y":0.413},"p2":{"x":0.932,"y":0.877}},"type":"rectangle"},
      {"data":[{"x":0.143,"y":0.807},{"x":0.317,"y":0.87},{"x":0.511,"y":0.145},{"x":0.328,"y":0.096},{"x":0.096,"y":0.554}],"type":"polygon"}] 
      Copied to clipboard
    4. If everything is OK, close the preview tab. If not, check the data you inserted in the code blocks.

    Note. You won't be able to create the interface of this project in Template Builder.
  5. Save the changes.

  6. Write instructions for Tolokers:

    Instructions:

    Look at the image and answer the question: Are all traffic signs outlined correctly?
    If there are, click Yes.
    If there isn't, click No.
    For example, road signs are outlined correctly, so the correct answer is Yes.
    Copied to clipboard
    Tip. If you want to add a task examples in the instruction, complete it yourself in the preview mode. Take screenshots, upload them to photo hosting or cloud storage and insert image links in the instructions by clicking the button on the toolbar.
  7. Click Finish.

Create a pool

  1. Open the project page titled Are the traffic signs outlined correctly?.

  2. Click the Add a pool button.

  3. Specify the Pool name.

  4. (optional) Add a Private comment. This information is available only to you.

  5. Filter Tolokers in the Tolokers section of the Audience block:

    1. Click Add filter.

    2. Add the Region by phone number and Languages filters and select Tolokers from Russia, Ukraine, Kazakhstan, and Belarus who indicated in their profiles that they know Russian.

    3. Add another filter. Click Add filter.

    4. Find the Skills block in the list and select the My skills skill.

    5. In the Skills field, select Area selection.

    6. In the ? field specify =.

    7. Leave the Missing field blank.

    Tasks in pools will automatically be available in the web version of Toloka and the mobile app. If you want to change the default settings and limit the visibility of the task for any of the versions, add the Client filter and select the desired value: Toloka web version or Toloka for mobile.

  6. (optional) In the Speed/quality balance section, specify the desired quality level. Improving quality may reduce the speed of task completion because the pool will be available to fewer Tolokers. Learn more in the Speed/quality balance section.

  7. In the Price block, find the Price per task suite field and specify the price. For example, 0.01.

  8. In the Quality control block, set up quality control settings for the pool:

    1. Click Add a quality control rule.

    2. Find the Rules block in the list and choose Majority vote.

    3. In the Accept as majority field, soecify 2.

    4. Set the rule: if number of responses ≥ 10 and correct responses (%) < 50, then ban the Toloker on project for 10 days. Specify the reason Doesn't match the majority.

      Note.

      The rule takes effect when the number of responses for the task is equal to the overlap.

    Learn more in Quality control.

  9. In the Task overlap section, enter 3 in the The number of Tolokers to complete each task field.

  10. In the Additional settings block:

    1. Enter 600 in the Time per task suite field.

    2. Turn on the Keep task order option.

  11. Click Create a pool.

Prepare and upload a TSV file with the results

  1. Prepare a TSV file with tasks.

    1. In the text or spreadsheet editor, open the TSV file that you received after aggregating the results in project 2.
    2. Leave the INPUT:image column unchanged.

    3. Change the OUTPUT:result column name to INPUT:selection.

    4. Rename the ASSIGNMENT:assignment_id column to INPUT:assignment_id.

    5. Delete the ACCEPT:verdict: and ACCEPT:comment: columns:

    6. Add input data, for example:
      [{""type"":""rectangle"",""data"":{""p1"":{""x"":0.2421,""y"":0.98871},""p2"":{""x"":0.93663,""y"":0.8776}}}]
      Copied to clipboard
      and save the file in TSV format.
    Tip.

    You can prepare TSV file in the Notepad program. To transfer data to Microsoft Excel, use Text Import Wizard and disable the Text qualifier option.

  2. Upload the resulting TSV file to the Are the traffic signs outlined correctly? pool:

    1. Open the Are the traffic signs outlined correctly? pool.

    2. Click Upload. In the window that opens, configure the file upload settings.

    3. Choose Set manually.

    4. In the Tasks per suite field, specify 10.

    5. Click Upload.

    6. In the window that opens, choose the TSV file with tasks to upload and click Open.

    7. In the window that opens, check the number of tasks and click Add.

    8. On the pool page, click Preview. Make sure that the task displays images with the selected objects.
      Tip.

      If there are no selected objects, check if the TSV file has quotation marks inserted correctly in the INPUT:selection column. Make sure that the interface parameters are set correctly.

  3. Click to start the pool.

    Attention.

    The tasks will be completed by real Tolokers in Toloka. Recheck your project setup before you start the pool.

Download the reviewed results

  1. Next to the Download results button, click .

  2. Choose Dawid-Skene aggregation model from the list. Learn more about Result aggregation based on the Dawid-Skene model.

  3. In the window that opens, click Yes.

  4. At the top of the page, click View the list of operations.

    Note.

    To track the progress, refresh the page from time to time. Aggregation takes from 5 to 20 minutes. You can start designing another project meanwhile.

  5. When the operation is complete, download the TSV file with the results. To do this, click Download in the Files column.

  6. Use the TSV file with the results in project 2.

Check the completed tasks

Since the Non-automatic acceptance option is enabled in the pool settings for project 2, you should check the Tolokers' responses within the time limit set in the Review period field.

You can check the results in two ways:

  • In the TSV file with the results.
  • In the pool interface.
  1. In the text or spreadsheet editor, open the TSV file that you received after aggregating the results.

  2. Prepare the TSV file:

    1. Add a column named ACCEPT:verdict with the review results.

    2. Add a column named ACCEPT:comment with comments for Tolokers if responses were rejected. For example, comment on which part of the instructions wasn't followed.

    3. Rename the INPUT:assignment_id column to ASSIGNMENT:assignment_id.

  3. Fill in the ACCEPT:verdict: and ACCEPT:comment: columns:

    • If the aggregate result of the assignment is OK, put + to accept it.
    • If the aggregate result of the assignment is incorrect or it doesn't open, put - to reject it. Enter the reason for rejecting the task in the ACCEPT:comment: field, for example, The object isn't selected or is selected incorrectly.
    Tip. You can use the awk commands to outline images on Linux and MacOS devices:
    awk 'BEGIN {FS=OFS="\t";} NR>1 {if($4~"OK"){ print $1, "+", ""; }else{ print $1, "-", "The object isn't selected or is selected incorrectly.";}}' <post_accept_res>.tsv > <review_res>.tsv
    Copied to clipboard
  4. Delete all the other columns.

  5. Save the file in TSV format.

  6. Open the pool page in project 2.

  7. Click Review assignments.

  8. Click Upload results.

  9. In the window that opens, choose the TSV file with the results to upload and click Open.

  10. In the window that opens, compare the number of tasks in the Processed successfully and Total submitted fields on the pool page.

  11. Click Add.

  12. In the window that opens, click Close.

  13. When setting up a pool in the second project you turned on the Recompletion of the rejected tasks option.

    In this case, the pool automatically reopens and the assignments are reassigned to other Tolokers. When they're completed, send the results for review. Then download the results, check them, and upload the reviewed results. You can reject assignments as many times as you want to get more accurate results.