ICWE 2022: web engineering with human-in-the-loop

In this tutorial we present a portion of our six-year experience in solving real-world tasks that combine efforts made by humans and machines.

Jul 8, 2022

Past conference

Overview

Modern Web applications employ sophisticated Machine Learning models to rank news, posts, products, and other items presented to the users or contributed by them. To keep these models useful, one has to constantly train, evaluate, and monitor these models using freshly annotated data, which can be done using crowdsourcing.

In this tutorial we present a portion of our six-year experience in solving real-world tasks with human-in-the-loop pipelines that combine efforts made by humans and machines. We introduce data labeling via public crowdsourcing marketplaces and present the critical components of efficient data labeling. Then, we run a practical session, where participants address a challenging real-world Information Retrieval for e-Commerce task, experiment with selecting settings for the labeling process, and launch their label collection project on real crowds within the tutorial session. We present useful quality control techniques and provide the attendees with an opportunity to discuss their annotation ideas. Methods and techniques described in this tutorial can be applied to any crowdsourced data and are not bound to any specific crowdsourcing platform.