consistency

crowdkit.metrics.data._classification.consistency | Source code

consistency(
answers: DataFrame,
workers_skills: Optional[Series] = None,
aggregator: BaseClassificationAggregator = ...,
by_task: bool = False
)

Consistency metric: posterior probability of aggregated label given workers skills

calculated using standard Dawid-Skene model.

Parameters description

ParametersTypeDescription
answersDataFrame

A data frame containing task, worker and label columns.

workers_skillsOptional[Series]

workers skills e.g. golden set skills. If not provided, uses aggregator's workers_skills attribute.

aggregatorBaseClassificationAggregator

aggregation method, default: MajorityVote

by_taskbool

if set, returns consistencies for every task in provided data frame.

  • Returns:

    Union[float, pd.Series]

  • Return type:

    Union[float, Series]

Last updated: March 31, 2023

Crowd-Kit
Overview
Reference
Aggregation
Datasets
Learning
Metrics
Postprocessing