alpha_krippendorff
crowdkit.metrics.data._classification.alpha_krippendorff | Source code
alpha_krippendorff(answers: DataFrame, distance: Callable[[Hashable, Hashable], float] = binary_distance)
Inter-annotator agreement coefficient (Krippendorff 1980).
Amount that annotators agreed on label assignments beyond what is expected by chance. The value of alpha should be interpreted as follows. alpha >= 0.8 indicates a reliable annotation, alpha >= 0.667 allows making tentative conclusions only, while the lower values suggest the unreliable annotation.
Parameters Description
| Parameters | Type | Description |
|---|---|---|
answers |
DataFrame | A data frame containing |
distance |
Callable[[Hashable, Hashable], float] | Distance metric, that takes two arguments, and returns a value between 0.0 and 1.0 By default: binary_distance (0.0 for equal labels 1.0 otherwise). |
-
Returns:
Float value.
-
Return type:
float
Examples:
Consistent answers.
alpha_krippendorff(pd.DataFrame.from_records([
{'task': 'X', 'worker': 'A', 'label': 'Yes'},
{'task': 'X', 'worker': 'B', 'label': 'Yes'},
{'task': 'Y', 'worker': 'A', 'label': 'No'},
{'task': 'Y', 'worker': 'B', 'label': 'No'},
]))
Partially inconsistent answers.
alpha_krippendorff(pd.DataFrame.from_records([
{'task': 'X', 'worker': 'A', 'label': 'Yes'},
{'task': 'X', 'worker': 'B', 'label': 'Yes'},
{'task': 'Y', 'worker': 'A', 'label': 'No'},
{'task': 'Y', 'worker': 'B', 'label': 'No'},
{'task': 'Z', 'worker': 'A', 'label': 'Yes'},
{'task': 'Z', 'worker': 'B', 'label': 'No'},
]))