Object recognition and area selection

  1. Create a project
  2. Add a task pool
  3. Upload tasks
  4. Set up quality control
  5. Start the pool and get the results
Tip.

Run the project in the Sandbox first. This helps you avoid making mistakes and spending money on a task that isn't working right.

Toloka provides an editor for selecting an image area. It lets the performer select a polygon or rectangle area.

Let's say you have many photos with animals and you need to select dogs and identify their breed. To do this, create a task: the performer is shown a photo and has to select the dog and specify its breed.

Example of a prepared task

To run tasks and get responses:

Create a project

The project defines what the task will look like for a performer.

  1. Choose a template:

    1. Click + Create project.

    2. Select the Object selection in an image template.
  2. Provide general information:

    1. Enter a clear name and a short description for the project. Performers will see this in the task list.

    2. Optionally add a Private comment.

    3. Click Save.

  3. Edit the task interface:

    Note. This tutorial shows how to create a task interface in the HTML/JS/CSS editor. You can also try creating a task interface in Template Builder.
    1. Define which objects you are going to pass to the performers and receive from them in response. To do this, add input and output fields in the Data specification section.

      What are input and output data?

      Input data is types of objects that are passed to the performer for completing the task. For example, this could be a text, an image, or geographic coordinates.

      Output data is types of objects that you receive after the task is completed. For example, this could be one of several response options, typed text, or an uploaded file.

      Learn more about input and output data fields.

      The template includes these fields:

      • The image input data field for a link to an image.

        Change the data type to string to add links to your files.

      • The result output data field with the "json" type for the JSON object with the point coordinates.

      Add the output data field — the required breed string to write the dog breed specified by the performer.

      Create the task interface in the HTML block. It describes how the task elements should be arranged in the task.

      You can use standard HTML tags and special expressions in double curly brackets for input and output data fields.

      {{field type="image-annotation" name="result" src=(proxy image)}}
      {{field type="input" name="breed" placeholder="Enter the dog's breed" width="100%"}}

      This notation describes the following task design:

      • A picture with the area selection tool.
      • Text input field.

      Leave the CSS and JavaScript blocks unchanged.

    2. Click to see the performer's view of the task.

      Note. The project preview shows one task with standard data. You can define the number of tasks to show on the page later.
    3. Save the changes.

  4. Write instructions for performers:

    1. Write short and clear guidelines (see the recommendations). Describe what needs to be done and give examples in them.

      You can prepare instructions in HTML format, then copy and paste into the editor. Click <> to switch to HTML mode.

    2. Click Finish.

Add a task pool

A pool is a set of paid tasks sent out for completion at the same time.

  1. Open the project and click Add pool.
  2. Give the pool any convenient name. It is available only to you — the performer will only see the name of the project.
  3. Set the price per task suite (for instance, $0.02).
    What is a task suite?

    A task suite can contain one or several tasks that are shown on the same page. If the tasks are simple, you can add 10-20 tasks per suite. Don't make task suites too long because it slows down loading speed for performers.

    Performers get paid for completing the entire task suite.

    The number of tasks per suite is set when uploading tasks.

    What is a fair price for a task suite?

    The general rule of pricing is the more time the performer spends to complete the task, the higher the price is.

    You can register in Toloka as a performer and find out how much other requesters pay for tasks, or see examples of cost for different types of tasks.

  4. Set the Time allowed for completing a task suite. It should be long enough to read the instructions and wait for task data to download. (for example, 1200 seconds).
  5. Set the Overlap, which is the number of performers to complete the same task. For image area selection tasks, it is usually 1.
  6. Turn on the Non-automatic acceptance option and enter the number of days for checking the task in the Deadline field (for example, 7).
    What is non-automatic acceptance (assignment review)?

    The non-automatic acceptance option allows you to review completed assignments before accepting them and paying for them. If the performer didn't follow instructions, you can reject the assignment. The maximum allowed period for the review is set in the Deadline field.

  7. Add Filters to choose performers.
  8. Save the pool.

Upload tasks

Prepare your own task file. Check out the example in the demo TSV file. In the Toloka interface under the pool name, there are links to TSV file templates for regular, control, and training tasks.

  1. Click Upload. In the window that opens, you can also download a sample TSV file by clicking Sample file for uploading tasks.
    What is TSV?
    A TSV file presents a table as a text file in which columns are separated by tabs.

    You can work with it both in a table editor and a text editor, and then save it to the desired format. More about working with a TSV file. There is a CSV format that is similar to TSV, but you should use a TSV file for uploading.

  2. Add input data in it. The header of the input data column contains the word INPUT. Leave the other columns empty.
  3. Upload the tasks: choose Set manually and set the number of tasks (for example, 4 tasks per page).

Set up quality control

Quality control rules allow you to filter out inattentive performers. You can configure quality control both in the project and in the pool.

Attention.

Quality control settings are applied to all project pools, so you can't change them in just one of the pools.

    Go to pool editing (the Edit button in the upper-right corner of the page) and click Add Quality Control Rule.

    You can copy quality control settings from another pool. To do this, click Copy settings from in the Users filter section.

  1. Add a restriction for Fast responses.

    The Minimum time per page value depends on the number of tasks on this page. It takes 2-4 seconds to identify the cat's mood. This means that a page with 10 tasks may take 20-30 seconds to complete.

    A performer can make an accidental mistake once in a while, but after 2-3 repeated mistakes you can ban the performer for a while.

    Specify the following values:

    This means that a user who submits two assignments in less than 20 seconds will be blocked for 10 days and won't be able to access your tasks.

    Attention. The assignments submitted by banned performers will be taken into account if they are not rejected manually using assignment review They can be reassigned by setting up the Recompletion of assignments from banned users rule.
  2. Add the Review results quality control rule and enter the following values:

    This means that if 35% or more of a performer's responses are rejected, the performer is banned and can't access your tasks for 15 days. The rule takes effect after 3 responses of the performer are reviewed.

Start the pool and get the results

  1. Start the pool by clicking .
  2. Track the completion of tasks in the Pool statistics section.
  3. When the first results are received, you can start the review . After the specified time period, all responses are automatically accepted, regardless of their quality.
    To review assignments, go to the pool and click Review assignments.
    Note. The point coordinates are saved in JSON.

Troubleshooting

Do I need to convert all the images in the task to the same size or can they be different?
You can use different image sizes.
How do I create a shortcut for adding a polygon in "image-annotation"?
To create a shortcut, add the following action to the "onKey" method:
onKey: function(key) {
    var el = this.getDOMElement().querySelector(".image-annotation-editor__shape-polygon");

    if (key === 'D') {
      el.click();
      el.classList.add('image-annotation-editor__shape_active')
    }
Copied to clipboard
How do I mark up triangles so that they close automatically when the third point is selected?

Use the C shortcut for closing areas.

You can also use the library to customize the keyboard shortcuts for your tasks.

How do I create a task for selecting objects in images?
We recommend that you break down your object selection project into three projects in Toloka:
  1. Sorting images containing an object.
    1. Create a task using the “Image classification” template.
    2. Sort the images containing the object you are looking for.
    3. Show the image to the performer and ask if the object is in the image. Response options: Yes/No.
  2. Selecting objects in images.
    1. Select the object in the images you obtained after the previous project. You already have such a project. Run the task with non-automatic acceptance.
    2. Use the quality control rules: fast responses, non-automatic acceptance, and post-review re-assessment. Description of rules with examples.
  3. Reviewing object selection assignments.
    1. Create a task using the object selection template.
    2. Hide the editor and ask whether the object is selected correctly. Response options: Yes/No.
    3. In the input data, pass the images and coordinates of the labeled objects from the previous task.
    Now you can run the resulting pool with an overlap of 3-5 or with dynamic overlap. After that, you can aggregate the results and then upload the data for review to Project 2.

    To prevent the users who worked on the second project from doing the review, assign a skill to them. Use this skill as a filter in the pools of the third project.

How do I add assignment validation depending on a checkbox?

You can use JavaScript to add assignment validation depending on a checkbox. An example is provided in the “Search for information online” template.

I have a task for area selection in an image. What should the performer do if there is no selectable object in the image?
Main options:
  • Select an arbitrary area in the image (for example, put a square in the upper-right corner). In this case, the project instructions for reviewers should also reflect this.
  • Ask the performer to skip the task and report it in a personal message. Messages are reviewed by the requester. If the object is truly missing, the task is deleted from the pool by resetting the overlap.
  • Add an additional “No object” checkbox to the interface. Make sure that your JS checks that either the object is selected or the checkbox is enabled. In this case, add information about the checkbox value in the review task interface.
How much would 2000 images with a large number of different types of selectable objects cost? How do I create a task for this amount of work?

In the case of crowdsourcing, it's better to break down this task. The simpler the task, the cheaper it is and the better the quality of the final result. The cost of labeling a single class of objects in photos might be about $0.01.

​Base your task on the “Object selection in an image” template. See the step-by-step guide for creating this type of project on this page.

The editor used in the template lets you add a drop-down list for labeling the selected object. See how to do this in the editor description (Dropdown list tab).

How do I implement selection of 3 different areas in an image? Select the name, image, and price in the product page screenshot.

You can create a selection + drop-down list with category selection. See how to do this in the editor description (Dropdown list tab).

What are the input data in the case of object labeling in an image: the coordinates of the object relative to the image, or the coordinates of the object in the Toloka user window?

The coordinates are relative to the image.

How do I use the control tasks in the standard template with an area selection editor?

In the standard template with an area selection editor, you can't use the control tasks, because in order for the assignment to be accepted by the system as correct, the object selected by the user must exactly match the control object. This is almost impossible. Therefore, you can leave the GOLDEN field empty in the task file or simply delete all the columns except INPUT.

How do I use control and training tasks in the standard template with an area selection editor?

In the standard template with an area selection editor, you can't use the control tasks, because in order for the assignment to be accepted by the system as correct, the object selected by the user must exactly match the control object. This is almost impossible. Therefore, you can leave the GOLDEN field empty in the task file or simply delete all the columns except INPUT.

You can't use training and the main pool with the Training type in an area selection project because for the response to be correct, the object selected by the user must exactly match the control object. This is almost impossible.

Such tasks are usually run with non-automatic acceptance: the performer submits an assignment, and then the assignment is rejected or accepted after the review.

For pre-selection of users, you can use “examination tasks”. Review the assignments and assign skills based on the percentage of accepted assignments. For this purpose, add the “Results of assignment review” rule to the pool. To make sure that only the good performers are admitted to the main pool, put a skill-based filter to the pool.