Crowd Science Workshop at NeurIPS 2020

Overview

Despite the clear advantages of AI, automation driven by machine learning carries pitfalls that affect the lives of millions of people. The negative repercussions include the disappearance of many well-established mass professions and increased consumption of labeled data produced by humans. This data is not always obtained in a positive environment: data suppliers are often managed in an old-fashioned way and have to work full-time on routine pre-assigned tasks, leading to job dissatisfaction. Crowdsourcing is a modern and effective alternative as it gives flexibility and freedom to task executors in terms of place, time and the task type they want to work on. However, many potential stakeholders of crowdsourcing processes hesitate to use this technology due to a series of doubts that have continued to circulate over the past decade. To address these issues, our workshop focuses on the research and industry communities and covers three important aspects of data supply: Remoteness, Fairness, and Mechanisms.

Remoteness.

Data labeling requesters (data consumers for ML systems) doubt the effectiveness and efficiency of remote work. They need trustworthy quality control techniques and ways to guarantee reliable results on time. Crowdsourcing is one of the viable solutions for effective remote work. However, despite the rapid growth and the body of literature on the topic, crowdsourcing is in its infancy and, to a large extent, is still an art. It lacks clear guidelines and accepted practices for both the requesters and the performers (also known as workers), which makes it much harder to reach the full potential of this technology. We intend to reverse this and achieve a breakthrough by turning the art into a science.

Fairness.

Crowd workers (data suppliers) doubt the availability and choice of tasks. They need fair and ethical task assignment, fair compensation, and growth opportunities. We believe that the working environment (e.g. a crowdsourcing platform) may help meet these needs — it should provide flexibility in choosing tasks and working hours, and access to tasks should be fair and ethical. We also aim to address bias in task design and execution that can skew results in ways that data requesters don’t anticipate. Since quality, fairness and growth opportunities for performers are central to our workshop, we invite a diverse group of performers from a global public crowdsourcing platform to our panel-led discussion.

Mechanisms.

Matchmakers (the working environment, usually represented by a crowdsourcing platform) doubt the effectiveness of economic mechanisms that underlie their two-sided market. They need a mechanism design that guarantees proper incentives for both sides: flexibility and fairness for workers, and quality and efficiency for data requesters. We stress that economic mechanisms are the key to address the issues of remoteness and fairness successfully. Our intention is to deepen the interaction between and within communities that work on mechanisms and crowdsourcing.

Speakers

Lora Aroyo

Google Research, USA

Gianluca Demartini

University of Queensland

Praveen Paritosh

Google Research, USA

Matt Lease

University of Texas at Austin

Seid Muhie Yimam

Universität Hamburg

Panel Discussion

Our panel discussion will gather all stakeholders: researchers, representatives of global crowd platforms like Toloka and Amazon MTurk, performers, and requesters who work with the crowd on a large scale. We hope to stimulate a fruitful conversation, shed light on what is not often discussed, develop solutions to problems and find new growth points for crowdsourcing.

Topics

Panelists

Schedule