Toloka documentation

accuracy_on_aggregates

crowdkit.metrics.workers.accuracy_on_aggregates.accuracy_on_aggregates | Source code

accuracy_on_aggregates(
    answers: DataFrame,
    aggregator: Optional[BaseClassificationAggregator] = ...,
    aggregates: Optional[Series] = None,
    by: Optional[str] = None
)

Accuracy on aggregates: a fraction of worker's answers that match the aggregated one.

Parameters Description

Parameters Type Description
answers DataFrame

a data frame containing task, worker and label columns.

aggregator Optional[BaseClassificationAggregator]

aggregation algorithm. default: MajorityVote

aggregates Optional[Series]

aggregated answers for provided tasks.

by Optional[str]

if set, returns accuracies for every worker in provided data frame. Otherwise, returns an average accuracy of all workers.

Returns -

Union[float, pd.Series]