Project 3. Are the bounding boxes correct?

In this project, Tolokers will determine if traffic signs were selected correctly on the images in project 1, and submit the responses in project 2.

Create a project

In the interface:

1. Choose a preset:

   1. Click Create a project.

      

   2. Click Do it myself.

   3. Select the Object recognition & detection preset.

2. In the General information section, add the project name and description:

   1. In the Name to show Tolokers field, enter Are the traffic signs outlined correctly?

   2. In the Description for Tolokers field, enter Look at the image and decide whether the traffic signs are outlined correctly or not.

   3. Optionally add a Private comment.

3. Edit the task interface. The output data format depends on the interface editor, so choose the same editor as in project 2.

   Template Builder

   HTML/CSS/JS editor

   1. Create a template based on the Checking the selected area example that uses Component for image labeling.

   2. To see the input and output data fields, click Show specifications in the Data specification section.

      Input data fields used in the project:

      • image — image address;

      • selection — an array with the selected area coordinates;

      • assignment_id — task ID.

      The Toloker's response will be recorded in the verdict output data field.

   3. Turn on the Define data specification manually option.

   4. Click to switch graphic mode to JSON format.

   5. Remove the template code from the Input data field and enter the following code:

      {
          "image": {
              "type": "url",
              "hidden": false,
              "required": true
          },
          "selection": {
              "type": "array_json",
              "hidden": false,
              "required": false
          },
          "assignment_id": {
              "type": "string",
              "hidden": true,
              "required": true
          }
      }
      Copy text

   6. Remove the template code from the Output data field and enter the following code:

      {
          "verdict": {
              "type": "string",
              "hidden": false,
              "required": true,
              "allowed_values": [
                  "OK",
                  "BAD"
              ]
          }
      }
      Copy text

4. In the Instructions for Tolokers editor, enter the instructions Tolokers will see when they start doing your tasks. You can add text, tables, and images to your instructions.

   Instructions text:

   Look at the image and answer the question: "Are all traffic signs outlined correctly?"
   If there are, click "Yes".
   If there isn't, click "No".
   For example, road signs are outlined correctly, so the correct answer is "Yes".
   Copy text

   Tip

   If you want to add a task examples in the instruction, complete it yourself in the preview mode. Take screenshots, upload them to photo hosting or cloud storage and insert image links in the instructions by clicking the button on the toolbar.

5. In the upper-right corner, click Save.

   Learn more about working with the project in the Project section.

Create a pool

1. Open the project page titled Are the traffic signs outlined correctly?.

2. Click Create new pool on the project page.

3. Select the value in the Pool type drop-down list.

   Pool types
   Note

   If the price per task suite is zero, you must select the pool type.

4. Set the Pool name (visible only to you) field. Only you will see this pool name on the project page.

5. Specify the pool description which will be displayed instead of the project description in the task list for Tolokers. By default, Tolokers see the description from the project settings. To use a different description, uncheck the Use project description box and set Public description. If necessary, click + Private comment to add a private project description that only you will see.

6. Click Create.

7. At the Select the audience for your task step, set up filters to select Tolokers for your pool.

   1. Use the Languages filter to select Tolokers with knowledge of the language.

   2. Clear My tasks may contain shocking or pornographic content if your project has none of those.

   3. Tasks in pools will be automatically available in the web version of Toloka and the mobile app. If you want to change the default settings and limit the visibility of the task for any of the versions, add the Client filter and select the desired value: Toloka web version or Toloka for mobile.

   4. Add another filter. Click Add filter.

   5. Find the Skills block in the list and select the My skills skill.

   6. In the Skills field, select Area selection.

   7. In the ? field specify =.

   8. Leave the Missing field blank.

8. (optional) In the Speed/quality balance section, specify the desired quality level. Improving quality may reduce the speed of task completion because the pool will be available to fewer Tolokers. Learn more in the Speed/quality balance section.

9. At the Set the task price and overlap step, set up how much a single task will cost for you.

   1. In Price per task suite, $, set the amount of money to pay per task suite done by one Toloker. For example, 0.01.

   2. Enter 3 in the Overlap field.

10. At the Setup quality control step, set quality control rules for more accurate results:

    1. Click Add a quality control rule.

    2. Find the Rules block in the list and choose Majority vote.

    3. In the Accept as majority field, soecify 2.

    4. Set the rule: if number of responses ≥ 10 and correct responses (%) < 50, then ban the Toloker on project for 10 days. Specify the reason Doesn't match the majority.

       Note

       The rule takes effect when the number of responses for the task is equal to the overlap.

    Learn more in Quality control.

11. At the Add optional pool settings step:

    1. At the Add optional pool settings step, specify the Time per task suite, sec.

       The time should be long enough to read the instructions and wait for task data to load. For example, 600 seconds.

12. At the Prepare and upload data step, upload your task data.

    1. Prepare a file with tasks.

       1. In the text or spreadsheet editor, open the file that you received after aggregating the results in project 2.

       2. Leave the INPUT:image column unchanged.

       3. Change the OUTPUT:result column name to INPUT:selection.

       4. Rename the ASSIGNMENT:assignment_id column to INPUT:assignment_id.

       5. Delete the ACCEPT:verdict: and ACCEPT:comment: columns.

       6. Save the file.

       Tip

       You can prepare the file in Notepad. To transfer data to Microsoft Excel, use Text Import Wizard and disable the Text qualifier option.

    2. Upload the file to the Are the traffic signs outlined correctly? pool:

       1. Open the Are the traffic signs outlined correctly? pool.

       2. Click Upload.

       3. Click Drop file here or select, and upload the file you’ve just made.

       4. Click Continue.

       5. Choose Set manually.

       6. In the Number of tasks per suite field, specify 10.

       7. Click Combine tasks into suites.

       8. On the pool page, click Preview. Make sure that the task displays images with the selected objects.

          Tip

          If there are no outlined objects, make sure that quotes are properly used in the INPUT:selection column in the file. Make sure that the interface parameters are set correctly.

13. Click to start the pool.

Download the reviewed results

1. Next to the Download results button, click .

2. Choose Dawid-Skene aggregation model from the list. Learn more about Result aggregation based on the Dawid-Skene model.

3. In the window that opens, click Yes.

4. At the top of the page, click View the list of operations.

   Note

   To track the progress, refresh the page from time to time. Aggregation takes from 5 to 20 minutes. You can start designing another project meanwhile.

5. When the operation is complete, download the results file. To do this, click Download in the Files column.

6. Use the results file in project 2.

Check the completed tasks

Since the manual review option is enabled in the pool settings for project 2, you should check the Tolokers' responses within the time limit set in the Review period field.

You can check the results in two ways.

What's next

• Learn more about decomposition.

Last updated: June 26, 2023