Toloka documentation


crowdkit.aggregation.classification.glad.GLAD | Source code

    n_iter: int = 100,
    tol: float = 1e-05,
    silent: bool = True,
    labels_priors: Optional[Series] = None,
    alphas_priors_mean: Optional[Series] = None,
    betas_priors_mean: Optional[Series] = None,
    m_step_max_iter: int = 25,
    m_step_tol: float = 0.01

Generative model of Labels, Abilities, and Difficulties.

A probabilistic model that parametrizes workers' abilities and tasks' dificulties. Let's consider a case of KK class classification. Let pp be a vector of prior class probabilities, αi(,+)\alpha_i \in (-\infty, +\infty) be a worker's ability parameter, βj(0,+)\beta_j \in (0, +\infty) be an inverse task's difficulty, zjz_j be a latent variable representing the true task's label, and yjiy^i_j be a worker's response that we observe. The relationships between this variables and parameters according to GLAD are represented by the following latent label model:

GLAD latent label model

The prior probability of zjz_j being equal to cc is

Pr(zj=c)=p[c],\operatorname{Pr}(z_j = c) = p[c],

the probability distribution of the worker's responses conditioned by the true label value cc follows the single coin Dawid-Skene model where the true label probability is a sigmoid function of the product of worker's ability and inverse task's difficulty:

Pr(yji=kzj=c)={a(i,j),k=c1a(i,j)K1,kc,\operatorname{Pr}(y^i_j = k | z_j = c) = \begin{cases}a(i, j), & k = c \\ \frac{1 - a(i,j)}{K-1}, & k \neq c\end{cases},


a(i,j)=11+exp(αiβj).a(i,j) = \frac{1}{1 + \exp(-\alpha_i\beta_j)}.

Parameters pp, α\alpha, β\beta and latent variables zz are optimized through the Expectation-Minimization algorithm.

J. Whitehill, P. Ruvolo, T. Wu, J. Bergsma, and J. Movellan. Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise. Proceedings of the 22nd International Conference on Neural Information Processing Systems, 2009

Parameters Description

Parameters Type Description
max_iter -

Maximum number of EM iterations.

eps -

Threshold for convergence criterion.

silent bool

If false, show progress bar.

labels_priors Optional[Series]

Prior label probabilities.

alphas_priors_mean Optional[Series]

Prior mean value of alpha parameters.

betas_priors_mean Optional[Series]

Prior mean value of beta parameters.

m_step_max_iter int

Maximum number of iterations of conjugate gradient method in M-step.

m_step_tol float

Tol parameter of conjugate gradient method in M-step.

labels_ Optional[Series]

Tasks' labels. A pandas.Series indexed by task such that labels.loc[task] is the tasks's most likely true label.

probas_ Optional[DataFrame]

Tasks' label probability distributions. A pandas.DataFrame indexed by task such that result.loc[task, label] is the probability of task's true label to be equal to label. Each probability is between 0 and 1, all task's probabilities should sum up to 1

alphas_ Series

workers' alpha parameters. A pandas.Series indexed by worker that contains estimated alpha parameters.

betas_ Series

Tasks' beta parameters. A pandas.Series indexed by task that contains estimated beta parameters.


from crowdkit.aggregation import GLAD
from crowdkit.datasets import load_dataset
df, gt = load_dataset('relevance-2')
glad = GLAD()
result = glad.fit_predict(df)

Methods Summary

Method Description
fit Fit the model through the EM-algorithm.
fit_predict Fit the model and return aggregated results.
fit_predict_proba Fit the model and return probability distributions on labels for each task.