HRRASA

crowdkit.aggregation.embeddings.hrrasa.HRRASA | Source code

HRRASA(
    self,
    n_iter: int = 100,
    tol: float = 1e-09,
    lambda_emb: float = 0.5,
    lambda_out: float = 0.5,
    alpha: float = 0.05,
    calculate_ranks: bool = False,
    output_similarity: Callable[[str, List[List[str]]], float] = glue_similarity
)
Copy text

Hybrid Reliability and Representation Aware Sequence Aggregation.

At the first step, the HRRASA estimates local workers reliabilities that represent how good is a worker's answer to one particular task. The local reliability of the worker $k$ on the task $i$ is denoted by $\gamma_i^k$ and is calculated as a sum of two terms:

The $\gamma_{i,emb}^k$ is a reliability calculated on embedding and the $\gamma_{i,seq}^k$ is a reliability calculated on output.

The $\gamma_{i,emb}^k$ is calculated by the following equation:

where $\mathcal{U_i}$ is a set of workers' responses on task $i$. The $\gamma_{i,out}^k$ makes use of some similarity measure $sim$ on the output data, e.g. GLUE similarity on texts:

The HRRASA also estimates global workers' reliabilities $\beta$ that are initialized by ones.

Next, the algorithm iteratively performs two steps:

1. For each task, estimate the aggregated embedding: $\hat{e}_i = \frac{\sum_k \gamma_i^k \beta_k e_i^k}{\sum_k \gamma_i^k \beta_k}$

2. For each worker, estimate the global reliability: $\beta_k = \frac{\chi^2_{(\alpha/2, |\mathcal{V}_k|)}}{\sum_i\left(\|e_i^k - \hat{e}_i\|^2/\gamma_i^k\right)}$, where $\mathcal{V}_k$ is a set of tasks completed by the worker $k$

Finally, the aggregated result is the output which embedding is the closest one to the $\hat{e}_i$. If calculate_ranks is true, the method also calculates ranks for each workers' response as

Jiyi Li. Crowdsourced Text Sequence Aggregation based on Hybrid Reliability and Representation. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’20), July 25–30, 2020, Virtual Event, China. ACM, New York, NY, USA,

https://doi.org/10.1145/3397271.3401239

Parameters Description

Examples:

import numpy as np
import pandas as pd
from crowdkit.aggregation import HRRASA
df = pd.DataFrame(
    [
        ['t1', 'p1', 'a', np.array([1.0, 0.0])],
        ['t1', 'p2', 'a', np.array([1.0, 0.0])],
        ['t1', 'p3', 'b', np.array([0.0, 1.0])]
    ],
    columns=['task', 'worker', 'output', 'embedding']
)
result = HRRASA().fit_predict(df)
Copy text

Methods Summary