Toloka documentation

ZeroBasedSkill

crowdkit.aggregation.classification.zero_based_skill.ZeroBasedSkill | Source code

ZeroBasedSkill(
    self,
    n_iter: int = 100,
    lr_init: float = 1.0,
    lr_steps_to_reduce: int = 20,
    lr_reduce_factor: float = 0.5,
    eps: float = 1e-05
)

The Zero-Based Skill aggregation model aka ZBS.

Performs weighted majority voting on tasks. After processing a pool of tasks, re-estimates workers' skills through a gradient descend step of optimization of the mean squared error of current skills and the fraction of responses that are equal to the aggregated labels.

Repeats this process until labels do not change or the number of iterations exceeds.

It's necessary that all workers in a dataset that send to 'predict' existed in answers the dataset that was sent to 'fit'.

Parameters Description

Parameters Type Description
n_iter int

A number of iterations to perform.

lr_init float

A starting learning rate.

lr_steps_to_reduce int

A number of steps necessary to decrease the learning rate.

lr_reduce_factor float

A factor that the learning rate will be multiplied by every lr_steps_to_reduce steps.

eps float

A convergence threshold.

Examples:

from crowdkit.aggregation import ZeroBasedSkill
from crowdkit.datasets import load_dataset
df, gt = load_dataset('relevance-2')
result = ZeroBasedSkill().fit_predict(df)

Methods Summary

Method Description
fit Fit the model.
fit_predict Fit the model and return aggregated results.
fit_predict_proba Fit the model and return probability distributions on labels for each task.
predict Infer the true labels when the model is fitted.
predict_proba Return probability distributions on labels for each task when the model is fitted.