Use decomposition
Break your task down into steps until each separate level is clear enough
for any performer to handle.
Get to the heart of crowdsourcing
Decomposition lies at the core of the crowdsourcing concept. It involves breaking a task down into parts: replace one large problem with a series of smaller, interconnected problems that are easier to solve and can be completed by different performers.

Benefits of decomposition

  • Reduce the number of errors
  • Make it easier to review the completed task
  • Lower the entry threshold for performers: the more accessible the task, the more people will complete it correctly and quickly
  • Ways to decompose a task

    1. Vertically

  • Divide a task into several projects (for example, to outline objects in images, first create a project for classifying the images in the dataset, and use a separate project for outlining the objects).
  • Divide a project into several steps (for example, classification followed by multi-classification).

    2. Horizontally

  • Use post-verification for checking tasks completed by other performers.
  • How do I know that a task needs to be decomposed?

    The rules here are fairly easy. If your task offers a choice of 3-5 answers and the instructions fit on one page without scrolling, then most likely your task doesn't need to be decomposed. In all other cases, you should probably try to break down the task.

    You can also discover when you need to decompose by running short experiments. If your task is taken up very slowly, all the performers fall out due to low skill levels, and the problem is not in your control tasks, you can assume that the task is too complex.

    Example: decomposing a labeling project

    Let's say you want to improve a computer vision algorithm that is used for self-driving cars. You give performers a set of photos of streets and ask them to outline every traffic light with a bounding box. After the labeling is finished you receive noisy labels. Why? Because the task didn’t make allowances for details such as photos without any traffic lights and you didn’t consider how to check the quality of results provided by thousands of anonymous performers.

    You can decompose this task and try again with a pipeline like this:

    1. Check whether the image contains traffic lights or not (classification)
    2. Outline each traffic light with a bounding box (object segmentation)
    3. Check whether traffic lights are outlined correctly (classification)

    To ensure quality, the second and third projects are completed by different performers. The third project checks whether the results of the second project are correct, and performers are only rewarded for correct results.

    To learn more about this pipeline, see the instructions.
    Yandex.Toloka News
    Receive information about platform updates, partners, training materials, and other news.
    Файлы cookies
    Для персонализации сервисов Яндекс использует файлы cookies. Продолжая использование сайта, вы соглашаетесь с этим. Подробности о файлах cookies и об обработке ваших данных в Политике конфиденциальности.
    Tue Nov 17 2020 21:51:27 GMT+0300 (Moscow Standard Time)