Overview

Overview

Aggregate accepted responses for tasks with an overlap in a pool of 2 or more to get the final response with the degree of confidence.

There are two ways to aggregate responses, depending on whether dynamic overlap is enabled in the pool (dynamic_overlap_config):
  • Yes — Aggregate responses by skill (WEIGHTED_DYNAMIC_OVERLAP).
  • No — Aggregate responses using the Dawid-Skene method (DAWID_SKENE).

    The Dawid-Skene aggregation model works with control and training tasks as well as with main tasks. Responses specified for control tasks might not match the actual responses to the task if performers often submit an incorrect response to it.

    The Dawid-Skene aggregation model automatically evaluates |L|² parameters for each performer, where L is the number of different aggregation values. Note that these parameters are determined automatically and are only used in calculations.

    Because the Dawid-Skene method evaluates |L|² parameters for each performer, we don't  recommend using it when the performer labels < |L|² tasks. In this case, the quality of aggregation may be poor.

    Aggregation via the interface and via the API works differently. When using the Dawid-Skene aggregation model, you can't aggregate more than one field via the API. You can only do that via the interface.

Methods

Method Overview End point
POST api/v1/aggregated-solutions/aggregate-by-pool

Starts aggregating responses to all completed tasks in the pool.

POST api/v1/aggregated-solutions/aggregate-by-task

Starts aggregating responses to a single task.

GET api/v1/aggregated-solutions/<operation_id>

Gets aggregated responses.

Method Overview End point
POST api/v1/aggregated-solutions/aggregate-by-pool

Starts aggregating responses to all completed tasks in the pool.

POST api/v1/aggregated-solutions/aggregate-by-task

Starts aggregating responses to a single task.

GET api/v1/aggregated-solutions/<operation_id>

Gets aggregated responses.