MMSR

crowdkit.aggregation.classification.m_msr.MMSR | Source code

MMSR(
    self,
    n_iter: int = 10000,
    tol: float = 1e-10,
    random_state: Optional[int] = 0,
    observation_matrix: ... = ...,
    covariation_matrix: ... = ...,
    n_common_tasks: ... = ...,
    n_workers: int = 0,
    n_tasks: int = 0,
    n_labels: int = 0,
    labels_mapping: Dict[Any, int] = ...,
    workers_mapping: Dict[Any, int] = ...,
    tasks_mapping: Dict[Any, int] = ...
)
Copy text

Matrix Mean-Subsequence-Reduced Algorithm.

The M-MSR assumes that workers have different level of expertise and associated with a vector of "skills" $\boldsymbol{s}$ which entries $s_i$ show the probability of the worker $i$ to answer correctly to the given task. Having that, we can show that

where $M$ is the total number of classes, $\widetilde{C}$ is a covariation matrix between workers, and $\boldsymbol{1}\boldsymbol{1}^T$ is the all-ones matrix which has the same size as $\widetilde{C}$.

So, the problem of recovering the skills vector $\boldsymbol{s}$ becomes equivalent to the rank-one matrix completion problem. The M-MSR algorithm is an iterative algorithm for rubust rank-one matrix completion, so its result is an estimator of the vector $\boldsymbol{s}$.

Then, the aggregation is the weighted majority vote with weights equal to $\log \frac{(M-1)s_i}{1-s_i}$.

Matrix Mean-Subsequence-Reduced Algorithm. Qianqian Ma and Alex Olshevsky. Adversarial Crowdsourcing Through Robust Rank-One Matrix Completion. 34th Conference on Neural Information Processing Systems (NeurIPS 2020)

https://arxiv.org/abs/2010.12181

Parameters Description

Examples:

from crowdkit.aggregation import MMSR
from crowdkit.datasets import load_dataset
df, gt = load_dataset('relevance-2')
mmsr = MMSR()
result = mmsr.fit_predict(df)
Copy text

Methods Summary