Toloka documentation

consistency

crowdkit.metrics.data._classification.consistency | Source code

consistency(
    answers: DataFrame,
    workers_skills: Optional[Series] = None,
    aggregator: BaseClassificationAggregator = ...,
    by_task: bool = False
)

Consistency metric: posterior probability of aggregated label given workers skills

calculated using standard Dawid-Skene model.

Parameters Description

Parameters Type Description
answers DataFrame

A data frame containing task, worker and label columns.
workers_skills Optional[Series]

workers skills e.g. golden set skills. If not provided, uses aggregator's workers_skills attribute.
aggregator BaseClassificationAggregator

aggregation method, default: MajorityVote
by_task bool

if set, returns consistencies for every task in provided data frame.

  • Returns:

    Union[float, pd.Series]

  • Return type:

    Union[float, Series]