Conference

RecSys 2022: Recommender systems with HITL

In our tutorial, we will share more than six years of our crowdsourcing experience and bridge the gap between crowdsourcing and recommender systems communities.

Sep 18, 2022, 21:30 UTC

+2

Overview

Today, most recommender systems employ Machine Learning to recommend posts, products, and other items, usually produced by the users. Although the impressive progress in Deep Learning and Reinforcement Learning, we observe that recommendations made by such systems still do not correlate with actual human preferences.

In our tutorial, we will share more than six years of our crowdsourcing experience and bridge the gap between crowdsourcing and recommender systems communities by showing how one can incorporate human-in-the-loop into their recommender system to gather the real human feedback on the ranked recommendations. We will discuss the ranking data lifecycle and run through it step-by-step. A significant portion of tutorial time is devoted to a hands-on practice, when the attendees will, under our guidance, sample and annotate recommendations on real crowds, build the ground truth dataset, and compute the evaluation scores.

All the demonstrated methodology is platform-agnostic and can be freely adapted to a variety of applications. One can gather the judgments on any data labeling platform, from in-house setups till MTurk and Toloka. A related tutorial was previously presented at NAACL-HLT '21, WWW '21, CVPR '20, SIGMOD '20, WSDM '20, and KDD '19.

We expect the attendees to understand the core concepts in recommender systems and are able to write short scripts in Python, while we do not require any knowledge of crowdsourcing. We will provide all the necessary definitions and icebreakers to accommodate a wider audience. We recommend the attendees to bring their laptops for the hands-on practice session.

Agenda

1. Introduction

- Recommender Systems
- Crowdsourcing
- Online and Offline Evaluation

2. Ranking and Its Quality

- Problem of Learning-to-Rank
- Pointwise/Pairwise/Listwise Approaches
- Evaluation Criteria

3. Human-in-the-Loop Essentials

- Core Concepts in Crowdsourcing
- Quality Control

4. Hands-On Practice Session

A hands-on practice, when the attendees will, under our guidance, sample and annotate recommendations on real crowds, build the ground truth dataset, and compute the evaluation scores.

5. From Human Labels to Ground Truth

- Problem of Answer Aggregation
- Pairwise Comparisons
- Crowd-Kit Library