Conference

Tutorial at SIGMOD 2020

In this tutorial, we present some key techniques for efficiently collecting labeled data, including aggregation, incremental relabeling, and dynamic pricing.

Image
Image
Image
+3
Image

Overview

In this tutorial, we introduce data labeling via public crowdsourcing marketplaces and present some key techniques for efficiently collecting labeled data , including aggregation, incremental relabeling, and dynamic pricing.

This is followed by a practice session, where participants choose one real label collection task, experiment with selecting settings for the labeling process, and launch their own label collection project on one of the largest crowdsourcing marketplaces. During the tutorial, all projects are run on the real Toloka crowd. While we are waiting for the crowd performers to annotate participants’ projects, we present the major theoretical results in efficient aggregation, incremental relabeling, and dynamic pricing. We also discuss the strengths and weaknesses of crowdsourcing, as well as applicability to real-world tasks, summarizing our five years of research and industrial expertise in crowdsourcing. All participants receive feedback on their projects and practical advice.

Speakers

Image
Alexey Drutsa
TolokaHead of Efficiency & Growth Division
Image
Valentina Fedorova
TolokaAnalyst
Image
Olga Megorskaya
TolokaCEO
Image
Evfrosiniya Zerminova
TolokaTechnical Product Manager
Image
Dmitry Ustalov
TolokaHead of Research
Image
Daria Baidakova
TolokaDirector of Educational Programs

Schedule

— The concept of crowdsourcing
— Crowdsourcing task examples
— Crowdsourcing platforms
— Yandex crowdsourcing experience

— Decomposition for an effective pipeline
— Task instruction & interface: best practices
— Quality control techniques

— How Toloka works
— Types of tasks in Toloka
— Creating a project in Toloka

— Dataset and required labels
— Discussion: how to collect labels?
— Data labeling pipeline for implementation

Participants:
— create
— configure
— run data labeling projects on real performers in real-time


— Aggregation models
— Incremental relabeling 
— Dynamic pricing

10:30 - 11:00

Break


— Completing the label collection process


— Project results
— Ideas for further work and research
— References to literature and other tutorials

Slides

Introduction
Part 1: Main components of data collection
Part 2: Toloka requester interface
Part 3: Brainstorming the pipeline
Part 4: Practical Session
Part 5: Theory on efficient aggregation
Part 6: Practical Session part 2
Part 7: Results and conclusions

Don't miss out

Be the first to hear about our workshops, 
tutorials, and webinars.
Fractal