Toloka research

Our approach

At Toloka, we are committed to unlocking AI opportunities. Every day, our researchers tackle pressing AI and ML challenges,
make appearances at prominent global events, and publish their findings in scientific journals. Scroll down to learn more.

Research papers

Browse through some of our latest work.

Data Labeling for Machine Learning Engineers: Project-Based Curriculum and Data-Centric Competitions
We describe the curriculum and course structure and show why and how we added elements of project-based learning and data-centric competitions to our course on crowdsourcing.
EAAI 2023
Findings of the WMT’22 Shared Task on Large-Scale Machine Translation Evaluation for African Languages
We present the results of the WMT’22 Shared Task on Large-Scale Machine Translation Evaluation for African Languages.
Read paperWMT 2022
Best Prompts for Text-to-Image Models and How to Find Them
We present a human-in-the-loop approach to learning the most useful combination of prompt keywords using a genetic algorithm.
Read paperNeurIPS 2022
Clustering Without Knowing How To: Application and Evaluation
Our experiments on two different image datasets, dresses from Zalando's FEIDEGGER and shoes from the Toloka Shoes Dataset, confirm that one can yield meaningful clusters with no machine learning algorithms purely with crowdsourcing.
Read paperHCOMP 2022

PreviousNext

Research powered by Toloka

Find out how Toloka powers top-tier research across the globe.

NTIRE 2022 Challenge
on Night Photography Rendering

The challenge solicited solutions that processed RAW camera images captured in night scenes. The organizers used Toloka to evaluate the visual appearance of the results. Mean opinion scores were calculated for a “people’s choice” ranking of solutions.

Conference

Machine Learning for Planetary Science

Labeling a large number of images can be labor- and time-consuming, and labeling images in planetary science often requires the help of crowdsourcing. In their “Machine Learning for Planetary Science” book NASA researchers acknowledge Toloka.

Book

AI for Good: Framework to Empower Digital Workers

Saiph Savage, an Assistant Professor at Northeastern University and Director of the Northeastern Civic AI Lab collaborated with Toloka to lead a research initiative called “A.I. For Good Framework to Empower Digital Workers” to help rural workers get better wages and conditions.

Project

WMT21 Collaboration with Toloka

As part of the conference, teams competed to devise the best system for machine translation of articles into different languages. Toloka provided human judgements as the ground truth for translation quality evaluation. All relevant language pairs were covered for fast evaluations using Toloka’s global crowd.

Conference

Citing Toloka in research papers

If you are using Toloka for research, please cite our work as follows: Pavlichenko N., Stelmakh I., and Ustalov D. CrowdSpeech and Vox DIY: Benchmark Dataset for Crowdsourced Audio Transcription. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks. 2021.

Top science events

We regularly hold tutorials and lead workshops at some of the biggest AI conferences around the globe.

Past conference

ECIR '23

Human-in-the-Loop learning for information retrieval.

Apr 6, 2023

Hosts:

WSDM '23

CANDLE: Collaboration of Humans and Learning Algorithms for Data Labeling.

Mar 3, 2023

Hosts:

Dmitry Ustalov

Saiph Savage

Niels van Berkel

Yang Liu

Ujwal Gadiraju

Past conference

NeurIPS '22

Toloka social at NeurIPS 2022. From old biases to new opportunities: Annotator empowerment and data excellence.

Dec 1, 2022

Hosts:

Maxim Kunakov

Andrew Braun

Elena Brandt

Peter Kocsis

Matteo Tafuro

Nathan Noiry

Past conference

RecSys '22

We shared more than six years of our crowdsourcing experience and bridge the gap between crowdsourcing and recommender systems communities.

Sep 18, 2022

Hosts:

CHI '22

This сrowd science workshop explores how a reimagined perspective on crowdsourcing platforms could provide a more equitable, fair, and rewarding experience.

Apr 30, 2022

Hosts:

VLDB '21

This сrowd science workshop focuses on the best practices for efficient and trustworthy crowdsourcing.

Aug 20, 2021

Hosts:

ICML '21

High-Quality Data Labeling at Scale with Toloka workshop aims to provide a comprehensive picture of how crowdsourcing can be applied to real life AI production.

Jul 18, 2021

Hosts:

Olga Megorskaya

Oleg Pavlov

Dmitry Ustalov

Omar Alonso

Saiph Savage

Daria Baidakova

Vladimir Losev

Past conference

NAACL-HLT '21

We shared some of the unique insights we have gained from six years of industry experience in efficient natural language data.

May 20, 2021

Hosts:

TheWebConf '21

In this tutorial, we presented a systematic view on using Human-in-the-Loop to obtain scalable offline evaluation processes and high-quality relevance judgements

Hosts:

Nikita Popov

Past conference

NeurIPS '20

At this сrowd science workshop we discussed key issues in preparing labeled data for machine learning, with a focus on remoteness, fairness, and mechanisms in the context of crowdso...

Dec 11, 2020

Hosts:

Lora Aroyo

Gianluca Demartini

Praveen Paritosh

Matt Lease

Seid Muhie Yimam

Past conference

CVPR '20

We presented a data processing pipeline used for training self-driving cars. Participants gained practical experience launching an annotation project in Toloka.

Hosts:

Ivan Semchuk

Denis Rogachevsky

Past conference

SIGMOD '20

We explored the practical aspects of how crowdsourcing can be applied to information retrieval. Participants learnt how to create a dataset with relevant products.

Hosts: