Crowd Science: Trust, Ethics, and Excellence in Crowdsourced Data Management at Scale. This workshop focuses on the best practices for efficient and trustworthy crowdsourcing.

In this introductory tutorial, we share some of the unique insights we have gained from six years of industry experience in efficient natural language data annotation via crowdsourcing.

In this talk, we will present the different quality control mechanisms for the collection of different NLP datasets.

We will discuss two major challenges in peer evaluation: (1) Fraud, where a group of participants form a coalition, get assigned to evaluate each other, and then give highly positive reviews; (2) Feedback, where opinions given by evaluatees (e.g., authors of papers) about evaluators (e.g., reviewers of papers) are biased by the evaluation given to them (e.g., whether the paper was accepted or rejected).

Weighted Majority Vote is a workhorse of crowdsourcing. But how do we assign these weights to different performers? In this talk, we will discuss a NeurIPS 2020 paper which theoretically answers this question!

Olga Megorskaya (Head of Toloka) and Igor Kuralenok (Head of ML Department, Yandex) will have a joint session at NVIDIA GTC. In their talk, they will dive deep into the pipeline of speech recognition, including GPU usage and the data annotation process. Register now to hear from them and other incredible speakers and discover the advanced technologies that are transforming today’s industries.

In our hands-on tutorial, we present a systematic view on using Human-in-the-Loop to obtain scalable offline evaluation processes and, in particular, high-quality relevance judgements. We will introduce the ranking problem to the attendees, discuss the commonly used ranking quality metrics, and then focus on Human-in-the-Loop-based approach to obtain relevance judgements at scale.

In this talk, we will reduce the skill-estimation problem to a statistical problem of matrix completion. Next, we will establish necessary and sufficient conditions under which efficient skill recovery is possible. Finally, we propose practical algorithms that carefully estimate the skills of the performers given their responses.

This talk presents the Social Media Mining for Health Applications (#SMM4H) shared tasks, and the first Russian adverse drug reaction corpus of tweets. We focus on current challenges and the imbalanced nature of datasets.

In this talk, we discuss several cases where advanced crowdsourcing, ranging from offline data collection to software testing, helps various business products – from search engines to self-driving cars.

In this talk, we discuss practical considerations for designing and implementing tasks that require the use of humans and machines in combination with the goal of producing high-quality labels.

We discuss several major challenges researchers face with conducting interactive studies online such as controlling for attention and drop-offs. We report our experience on running behavioral studies on Toloka crowdsourcing platform.

Will BERT kill data labeling? In recent years, large-scale unsupervised models like BERT and GPT-3 have made impressive progress with textual data, from text generation to question-answering and more. 

This talk presents principled approaches towards designing and evaluating large human decision-making systems. We focus on how to control sources of noisy data, like individual mistakes, dishonesty, and bias.

In this talk, we address the general problem of algorithmic and AI bias and identify known solutions, with a targeted discussion of the challenges inherent to crowdsourcing.

Top experts from the Toloka team discuss selected papers on crowd science from the NeurIPS conference and share their perspective on crowd research trends.

This talk introduces RuBQ, the first Russian-language Knowledge Base Question Answering (KBQA) dataset. Vladislav Korablinov describes the efficient data-collection pipeline designed in his work and shares valuable insights.

This talk focuses on an essential function for conversational agents: evaluating humor. We present an evaluation of 30 dialogue jokes to demonstrate that crowdsourcing is a viable option for humor evaluation and examine variation in humor judgments.

During this meetup, we discussed how the new generation of methods and tools allows the collection of high-quality human-labeled data on a large scale and why every ML specialist should know how to use crowdsourcing.

In this tutorial, “Crowdsourcing Practice for Efficient Data Labeling,” leading researchers and engineers share their industry expertise and research insights on key components of crowdsourcing, including efficient aggregation, incremental relabeling, and dynamic pricing.

We explore the practical aspects of how crowdsourcing can be applied to information retrieval. Learn how to create a dataset with relevant products.