Toloka research

Our approach

At Toloka, we are committed to unlocking AI opportunities. Every day, our researchers tackle pressing AI and ML challenges,
make appearances at prominent global events, and publish their findings in scientific journals. Scroll down to learn more.

Research papers

Browse through some of our latest work.

IMDB-WIKI-SbS: An Evaluation Dataset for Crowdsourced Pairwise Comparisons
Our dataset has balanced distributions of age and gender using the well-known IMDB-WIKI dataset as ground truth. We describe how our dataset is built and then compare several baseline methods, indicating its suitability for model evaluation.
IMDB-WIKI-SbS: An Evaluation Dataset for Crowdsourced Pairwise Comparisons Learn moreNeurIPS 2021
CrowdSpeech & Vox DIY: Benchmark Dataset for Crowdsourced Audio Transcription
Domain-specific data is the crux of the successful transfer of machine learning systems from benchmarks to real life. Crowdsourcing has become one of the standard tools for cheap and time-efficient data collection for simple problems such as image classification: thanks in large part to advances in research on aggregation methods.
CrowdSpeech & Vox DIY: Benchmark Dataset for Crowdsourced Audio Transcription Learn moreNeurIPS 2021
A General-Purpose Crowdsourcing Computational Quality Control Toolkit for Python
In this paper, we demonstrate Crowd-Kit, a general-purpose crowdsourcing computational quality control toolkit. It provides efficient implementations in Python of computational quality control algorithms for crowdsourcing, including uncertainty measures and crowd consensus methods.
A General-Purpose Crowdsourcing Computational Quality Control Toolkit for Python Learn moreHCOMP 2021
VLDB 2021 Crowd Science Challenge on Aggregating Crowdsourced Audio Transcriptions
This paper reviews the crowdsourced audio transcription shared task devoted to this problem and co-organized with the Crowd Science Workshop at VLDB 2021.
VLDB 2021 Crowd Science Challenge on Aggregating Crowdsourced Audio Transcriptions Learn moreVLDB 2021
Prediction of Hourly Earnings and Completion Time on a Crowdsourcing Platform
We study the problem of predicting future hourly earnings and task completion time for a crowdsourcing platform user who sees the list of available tasks and wants to select one of them to execute.
Prediction of Hourly Earnings and Completion Time on a Crowdsourcing Platform Learn moreKDD 2020
Text Recognition Using Anonymous CAPTCHA Answers
In this paper, we address the problem of labeling text images via CAPTCHA, where user identification is generally impossible. We propose a new algorithm to aggregate multiple guesses collected through CAPTCHA.
Text Recognition Using Anonymous CAPTCHA Answers Learn moreWSDM 2020

Citing Toloka in research papers

If you are using Toloka for research, please cite our work as follows:

Anna Lioznova, Alexey Drutsa, Vladimir Kukushkin, and Anastasia Bezzubtseva. 2020. Prediction of Hourly Earnings and Completion Time on a Crowdsourcing Platform. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD '20). Association for Computing Machinery, New York, NY, USA, 3172–3182. https://doi.org/10.1145/3394486.3403369

Top science events

We regularly hold tutorials and lead workshops at some of the biggest AI conferences around the globe.

Past conference

CHI '22

This сrowd science workshop explores how a reimagined perspective on crowdsourcing platforms could provide a more equitable, fair, and rewarding experience.

Apr 30, 2022

Past conference

VLDB '21

This сrowd science workshop focuses on the best practices for efficient and trustworthy crowdsourcing.

Aug 20, 2021

Hosts:

ICML '21

High-Quality Data Labeling at Scale with Toloka workshop aims to provide a comprehensive picture of how crowdsourcing can be applied to real life AI production.

Jul 18, 2021

Hosts:

Olga Megorskaya

Omar Alonso

Saiph Savage

Daria Baidakova

Dmitry Ustalov

Vladimir Losev

Oleg Pavlov

Past conference

NAACL-HLT '21

We shared some of the unique insights we have gained from six years of industry experience in efficient natural language data

May 20, 2021

Past conference

TheWebConf '21

In this tutorial, we presented a systematic view on using Human-in-the-Loop to obtain scalable offline evaluation processes and high-quality relevance judgements

Apr 13, 2021

Hosts:

Alexey Drutsa

Dmitry Ustalov

Daria Baidakova

Nikita Popov

Past conference

NeurIPS '20

On this сrowd science workshop we discussed key issues in preparing labeled data for machine learning, with a focus on remoteness, fairness, and mechanisms in the context of crowdsourcing for data collection and labeling.

Dec 11, 2020

Past conference

CVPR '20

We presented a data processing pipeline used for training self-driving cars. Participants gained practical experience launching an annotation project in Toloka.

Jun 15, 2020

Past conference

SIGMOD '20

We explored the practical aspects of how crowdsourcing can be applied to information retrieval. Participants learnt how to create a dataset with relevant products.

Jun 14, 2020

Past conference

WSDM '20

We explored the practice of efficient data collection via crowdsourcing: aggregation, incremental relabeling, and pricing.

Feb 3, 2020

Past conference

KDD '19

We introduced data labeling via public crowdsourcing marketplaces and will presented the key components of efficient label collection.

Aug 8, 2019