Toloka documentation

Wawa

crowdkit.aggregation.classification.wawa.Wawa | Source code

Wawa(self)

Worker Agreement with Aggregate.

This algorithm does three steps:

  1. Calculate the majority vote label
  2. Estimate workers' skills as a fraction of responses that are equal to the majority vote
  3. Calculate the weigthed majority vote based on skills from the previous step

Parameters Description

Parameters Type Description
labels_ Optional[Series]

Tasks' labels. A pandas.Series indexed by task such that labels.loc[task] is the tasks's most likely true label.

skills_ Optional[Series]

workers' skills. A pandas.Series index by workers and holding corresponding worker's skill

probas_ Optional[DataFrame]

Tasks' label probability distributions. A pandas.DataFrame indexed by task such that result.loc[task, label] is the probability of task's true label to be equal to label. Each probability is between 0 and 1, all task's probabilities should sum up to 1

Examples:

from crowdkit.aggregation import Wawa
from crowdkit.datasets import load_dataset
df, gt = load_dataset('relevance-2')
result = Wawa().fit_predict(df)

Methods Summary

Method Description
fit Fit the model.
fit_predict Fit the model and return aggregated results.
fit_predict_proba Fit the model and return probability distributions on labels for each task.
predict Infer the true labels when the model is fitted.
predict_proba Return probability distributions on labels for each task when the model is fitted.