Crowd-Kit Documentation

alpha_krippendorff

crowdkit.metrics.data._classification.alpha_krippendorff | Source code

alpha_krippendorff(answers: DataFrame, distance: Callable[[Hashable, Hashable], float] = binary_distance)

Inter-annotator agreement coefficient (Krippendorff 1980).

Amount that annotators agreed on label assignments beyond what is expected by chance. The value of alpha should be interpreted as follows. alpha >= 0.8 indicates a reliable annotation, alpha >= 0.667 allows making tentative conclusions only, while the lower values suggest the unreliable annotation.

Parameters Description

Parameters Type Description
answers DataFrame

A data frame containing task, worker and label columns.
distance Callable[[Hashable, Hashable], float]

Distance metric, that takes two arguments, and returns a value between 0.0 and 1.0 By default: binary_distance (0.0 for equal labels 1.0 otherwise).

  • Returns:

    Float value.

  • Return type:

    float

Examples:

Consistent answers.

alpha_krippendorff(pd.DataFrame.from_records([
    {'task': 'X', 'worker': 'A', 'label': 'Yes'},
    {'task': 'X', 'worker': 'B', 'label': 'Yes'},
    {'task': 'Y', 'worker': 'A', 'label': 'No'},
    {'task': 'Y', 'worker': 'B', 'label': 'No'},
]))

Partially inconsistent answers.

alpha_krippendorff(pd.DataFrame.from_records([
    {'task': 'X', 'worker': 'A', 'label': 'Yes'},
    {'task': 'X', 'worker': 'B', 'label': 'Yes'},
    {'task': 'Y', 'worker': 'A', 'label': 'No'},
    {'task': 'Y', 'worker': 'B', 'label': 'No'},
    {'task': 'Z', 'worker': 'A', 'label': 'Yes'},
    {'task': 'Z', 'worker': 'B', 'label': 'No'},
]))