Conference

RecSys 2022: Recommender systems with HITL

In our tutorial, we will share more than six years of our crowdsourcing experience and bridge the gap between crowdsourcing and recommender systems communities.

Image
Image
Image
+2
Image

Overview

Today, most recommender systems employ Machine Learning to recommend posts, products, and other items, usually produced by the users. Although the impressive progress in Deep Learning and Reinforcement Learning, we observe that recommendations made by such systems still do not correlate with actual human preferences.

In our tutorial, we will share more than six years of our crowdsourcing experience and bridge the gap between crowdsourcing and recommender systems communities by showing how one can incorporate human-in-the-loop into their recommender system to gather the real human feedback on the ranked recommendations. We will discuss the ranking data lifecycle and run through it step-by-step. A significant portion of tutorial time is devoted to a hands-on practice, when the attendees will, under our guidance, sample and annotate recommendations on real crowds, build the ground truth dataset, and compute the evaluation scores.

All the demonstrated methodology is platform-agnostic and can be freely adapted to a variety of applications. One can gather the judgments on any data labeling platform, from in-house setups till MTurk and Toloka. A related tutorial was previously presented at NAACL-HLT '21, WWW '21, CVPR '20, SIGMOD '20, WSDM '20, and KDD '19.

We expect the attendees to understand the core concepts in recommender systems and are able to write short scripts in Python, while we do not require any knowledge of crowdsourcing. We will provide all the necessary definitions and icebreakers to accommodate a wider audience. We recommend the attendees to bring their laptops for the hands-on practice session.

Agenda

1. Introduction

- Recommender Systems
- Crowdsourcing
- Online and Offline Evaluation

2. Ranking and Its Quality

- Problem of Learning-to-Rank
- Pointwise/Pairwise/Listwise Approaches
- Evaluation Criteria

3. Human-in-the-Loop Essentials

- Core Concepts in Crowdsourcing
- Quality Control

4. Hands-On Practice Session

A hands-on practice, when the attendees will, under our guidance, sample and annotate recommendations on real crowds, build the ground truth dataset, and compute the evaluation scores.

5. From Human Labels to Ground Truth

- Problem of Answer Aggregation
- Pairwise Comparisons
Crowd-Kit Library

6. Conclusion

- Discussion of Results
- References

Speakers

Image
Fedor Zhdanov
TolokaHead of MLProfile link
Image
Dmitry Ustalov
TolokaHead of ResearchProfile link
Image
Natalia Fedorova
TolokaEducation Program ManagerProfile link
Image
Nikita Pavlichenko
TolokaMachine Learning ResearcherProfile link
Image
Maxim Kunakov
TolokaCrowd Solution ArchitectProfile link

Slides

Part I: Introduction
Part II: Ranking and its Quality
Part III: Human-in-the-Loop Essentials
Part IV: Hands-On Practice Session
Part V: From Human Labels to Ground Truth

Similar events

Don't miss out

Be the first to hear about our workshops, 
tutorials, and webinars.
Fractal