consistency

crowdkit.metrics.data._classification.consistency | Source code

consistency(
    answers: DataFrame,
    workers_skills: Optional[Series] = None,
    aggregator: BaseClassificationAggregator = ...,
    by_task: bool = False
)

Consistency metric: posterior probability of aggregated label given workers skills

calculated using standard Dawid-Skene model.

Parameters Description

ParametersTypeDescription
answersDataFrame

A data frame containing task, worker and label columns.

workers_skillsOptional[Series]

workers skills e.g. golden set skills. If not provided, uses aggregator's workers_skills attribute.

aggregatorBaseClassificationAggregator

aggregation method, default: MajorityVote

by_taskbool

if set, returns consistencies for every task in provided data frame.

  • Returns:

    Union[float, pd.Series]

  • Return type:

    Union[float, Series]

Crowd-Kit
Overview
Reference
Aggregation
Datasets
Learning
Metrics
Postprocessing