As part of this report, we will consider what crowdsourcing is as an approach. We will also look at examples of data labeling pipelines and share how to reproduce and create new sources of marked up data.

In this talk, we focus on crowdsourced clustering and how to aggregate diverse opinions to find a consensus clustering solution in the most efficient way.

This talk describes collecting the first publicly available dataset of crowdsourced audio transcriptions and running a shared task data at the VLDB 2021 Crowd Science Workshop!

Projects that heavily rely on Python features such as metaclasses, decorators, or code-generation face a problem of poor support by static code analysis tools.

This talk introduces feet-on-street crowdsourcing, also known as spatial crowdsourcing, which implies performing the given task in a real-world location.

ML models are used in pathology to make advances in diagnosis and treatment. But there’s a major obstacle: it requires obtaining large high-resolution datasets.

We define a set of sensitive topics that can yield inappropriate and toxic messages and describe the methodology of collecting and labelling a dataset for appropriateness.

Olga Megorskaya discusses several cases where advanced crowdsourcing, ranging from offline data collection to software testing, plays an essential role in various business products — from search engines to self-driving cars.

Will BERT kill data labeling? - In recent years, large-scale unsupervised models like BERT and GPT-3 have made impressive progress with textual data, from text generation to question-answering and more.

This talk presents principled approaches towards design and evaluation of large human decision-making systems, focusing on important aspects of noise, strategic behaviour and bias.

Top experts from the Toloka team discuss selected papers on crowd science from the NeurIPS conference and share their perspective on crowd research trends.

We present an evaluation of 30 dialogue jokes to demonstrate that crowdsourcing is a viable option for humor evaluation and examine variation in humor judgments.

During this meetup, we talk about how the new generation of methods and tools allows to collect high quality human labeled data on a large scale and why every ML specialist should know how to use crowdsourcing.

We spoke about why static code analysis fails and discussed stub files: what are they and how can they help. We also shared how we solved static analysis issues with automatic stub generation.

Projects that heavily rely on such Python features as metaclasses, decorators, or code-generation face a problem of poor support by static code analysis tools. We talk about how we addressed this issue in our open-source projects.

This talk presents our work examining the current state of reporting of crowdsourcing experiments and offers guidance to address associated reporting issues.

We present the different quality control mechanisms considered for the collection of different NLP datasets using the well-known MTurk crowdsourcing and an “in-house” built social media-based crowdsourcing tool.

We discuss two major challenges in peer evaluation: fraud and feedback.

We present a novel analysis of the expected risk of weighted majority vote in multiclass classification and provide a specialized version of our bound for binary classification.

We reduce the skill-estimation problem to a statistical problem of matrix completion and propose practical algorithms that carefully estimate the skills of the performers given their responses.

We describe three tasks on mining adverse drug effects using annotated datasets, focusing on current challenges and the imbalanced nature of the datasets.

We discuss practical considerations for designing and implementing tasks that require the use of humans and machines in combination with the goal of producing high-quality labels.

We demonstrate how to run both asynchronous studies without real-time interactions between participants and synchronous, interactive studies where the partner’s decisions are delivered to participants in real time.

We describe the general problem of algorithmic and AI bias and some solutions that have been identified and . zoom in on crowdsourcing and provide a specialized discussion of the problems and their proposed solutions

We talk about how the new generation of methods and tools allows to collect high quality human labelled data on a large scale and why every ML specialist should know how to use crowdsourcing.

In this talk, we discuss how the new generation of methods and tools allows to collect high-quality human labeled data on a large scale and why every ML specialist should know how to use crowdsourcing.

Our course at the Conférence universitaire de Suisse occidentale introduces crowdsourcing as a practical methodology and helps participants master the essential steps and techniques to ensure top-quality data.

Together with Y-DATA in Tel Aviv, we created a 6-week community course taught by field experts. Students are exposed to both theory and practice, and apply their new knowledge in 5 full-cycle hands-on projects.

We introduce data labeling via public crowdsourcing marketplaces and present the key techniques for efficiently collecting labeled data. In the practice session, participants launch their own labeling project.

In this tutorial we share our unique industry experience in achieving efficient natural language annotation with crowdsourcing. In the practice session, participants launch their own labeling project.