Project 2. Select an object in the image

In this project, Tolokers will select image areas that contain a traffic sign. Use the results from project 1 as source images.

Create a project

In the interface:

1. Choose a preset:

   1. Click Create a project.

      

   2. Click Do it myself.

   3. Select the Object recognition & detection preset.

2. In the General information section, add the project name and description:

   1. In the Name to show Tolokers field, enter Outline traffic signs in the image.

   2. In the Description for Tolokers field, enter Draw a box around all traffic signs in the image.

   3. Optionally add a Private comment.

3. Edit the task interface in the editor you selected:

   Template Builder

   HTML/CSS/JS editor

   1. You can use it for this project with pre-configured validation and task layout.

      The Toloker won't be able to submit a response without selecting an area in an image.

      For more information about the template and its settings, see Object selection in an image in the Template Builder Help.

   2. To see the input and output data fields, click Show specifications in the Data specification section.

      • Input data field: image — A link to an image.

      • Output data field: result — The field that will contain the information about the labeled image after it is uploaded.

4. In the Instructions for Tolokers editor, enter the instructions Tolokers will see when they start doing your tasks. You can add text, tables, and images to your instructions.

   Instructions text: "Click and draw a rectangle around all traffic signs in the image".

   Tip

   If you want to add a task examples in the instruction, complete it yourself in the preview mode. Take screenshots, upload them to photo hosting or cloud storage and insert image links in the instructions by clicking the button on the toolbar.

5. In the upper-right corner, click Save.

   Learn more about working with the project in the Project section.

Create a pool

A pool is a set of paid tasks sent out for completion at the same time.

1. Open the project page Outline the traffic signs in the image.

2. Click Create new pool on the project page.

3. Select the value in the Pool type drop-down list.

   Pool types
   Note

   If the price per task suite is zero, you must select the pool type.

4. Set the Pool name (visible only to you) field. Only you will see this pool name on the project page.

5. Specify the pool description which will be displayed instead of the project description in the task list for Tolokers. By default, Tolokers see the description from the project settings. To use a different description, uncheck the Use project description box and set Public description. If necessary, click + Private comment to add a private project description that only you will see.

6. Click Create.

7. At the Select the audience for your task step, set up filters to select Tolokers for your pool.

   1. Use the Languages filter to select Tolokers with knowledge of the language.

   2. Clear My tasks may contain shocking or pornographic content if your project has none of those.

   3. Tasks in pools will be automatically available in the web version of Toloka and the mobile app. If you want to change the default settings and limit the visibility of the task for any of the versions, add the Client filter and select the desired value: Toloka web version or Toloka for mobile.

   4. Add a new skill. Click Add skill.

   5. In the window that opens, enter Area selection in the Title field.

   6. Click Add.

8. (optional) In the Speed/quality balance section, specify the desired quality level. Improving quality may reduce the speed of task completion because the pool will be available to fewer Tolokers. Learn more in the Speed/quality balance section.

9. At the Set the task price and overlap step, set up how much a single task will cost for you.

   1. In Price per task suite, $, set the amount of money to pay per task suite done by one Toloker. For example, 0.01.

      What is a task suite?
      What is a fair price for a task suite?

   2. In the Overlap field, define how many Tolokers must do each task.

      For image area selection tasks, it is usually 1.

10. At the Setup quality control step, set quality control rules for more accurate results:

    1. Turn on the Review task responses manually option.

       What is manual review?

       In the Review period in days field, specify the number of days for checking the task.

    2. Add the following quality control rules:

       • Recompletion of rejected assignments — sends the tasks you rejected to other Tolokers according to the specified rules.

         1. Click Add a quality control rule.

         2. Find the Rules block in the list and select Processing rejected and accepted assignments.

         3. Set a rule for a rejected task: if assignment becomes rejected, then extend overlap by 1. Turn on the Open pool if closed option.

            This means that the rejected task will be returned to the pool and shown to another Toloker.

       • Submitted responses — allows you to limit the number of pool tasks available for the Toloker per day.

         1. Click Add a quality control rule.

         2. Find Rules block in the list and choose Submitted responses.

         3. Set a rule for the completed task: if submitted assignments ≥ 1, then assign skill value Area selection equal to 1.

            These parameters allow you to mark the Toloker who completed at least one task in the pool.

            Tip

            If the Area selection skill isn't displayed in the list, save the pool and reopen it for editing.

       • Fast responses —Filters out Tolokers who respond too quickly.

         1. Click Add a quality control rule.

         2. Find the Rules block in the list and choose Fast responses.

         3. In the Recent tasks suites to use field, enter the number of recent task suites submitted by the Toloker. For example, 5.

         4. In the Minimum time per task suite field, enter the time in seconds. For example, 20.

         5. Set a rule for a fast response: if the number of fast responses ≥ 1, then ban on requester for 10 days. In the Reason field, enter Fast responses.

            This means that a user who completes a task suite in less than 20 seconds will be suspended for 10 days and won't be able to access your tasks.

       • Results of manual review — restricts the pool access for Tolokers who often make mistakes.

         1. Click Add a quality control rule.

         2. Find the Rules block in the list and choose Results of manual review.

         3. Set a rule for rejected tasks: if total reviewed responses ≥ 3 and rejected responses (%) > 35, then ban on requester for 15 days.

            This means that if 35% or more of a Toloker's responses are rejected, the Toloker is banned and can't access your tasks for 15 days. The rule takes effect after 3 responses of the Toloker are reviewed.

       Control tasks and majority vote aren't used for this project type, since the area markup provided by the Tolokers must match (which is almost impossible). Learn more in Quality control.

       Note

       You can copy quality control settings from another pool. To do this, in the Tolokers section of the Audience block, click copy them from another pool.

11. At the Add optional pool settings step, specify the Time per task suite, sec.

    The time should be long enough to read the instructions and wait for task data to load. For example, 1200 seconds.

12. At the Prepare and upload data step, upload your task data.

    1. Prepare a file with tasks.

       1. In the text or spreadsheet editor, open the file that you received after aggregating the results in project 1.

       2. Choose images appropriate for the current project (the OK value).

          Tip

          You can use the awk commands to outline images on Linux and MacOS devices:

          awk 'BEGIN {OFS = FS = "\t";} $2=/OK/ {print $1}' <aggregated_res>.tsv > <filtered_res>.tsv
          Copy text

       3. Copy the column with the selected links to a new sheet or document.

       4. Name it INPUT:image. If you want to set a different name, rename the column in the source file with the results, too.

       5. Save the file.

    2. Upload the tasks file:

       1. Click Drop file here or select, and upload the file you’ve just made.

       2. Click Continue.

       3. Choose Set manually.

       4. In the Number of tasks per suite field, specify 1.

       5. Click Combine tasks into suites.

13. At the Double-check your project and try out tasks step, check how the task will look from the Toloker's point of view.

    Note

    This step will be enabled after you complete the previous steps. You can skip this step by clicking Do it later.

14. Click to start the pool.

Get the results

1. On the pool page, click Download results. In the window that opens:

   1. In Status leave only the Submitted option enabled.

   2. In Columns leave only the assignment ID option enabled.

   3. Disable the Separate assignments with empty row option.

   4. Click Download results.

2. Use the results file in project 3.

What's next

• Create Project 3 to review assignments.

Last updated: June 26, 2023