ICWE 2022: Web Engineering with Human-in-the-Loop

In this tutorial we will present a portion of our six-year experience in solving real-world tasks with human-in-the-loop pipelines that combine efforts made by humans and machines.

Past conference
Don't miss the tutorial,
register right now
Register
Don't miss the tutorial,
register right now
Register

Overview

Modern Web applications employ sophisticated Machine Learning models to rank news, posts, products, and other items presented to the users or contributed by them. To keep these models useful, one has to constantly train, evaluate, and monitor these models using freshly annotated data, which can be done using crowdsourcing.

In this tutorial we will present a portion of our six-year experience in solving real-world tasks with human-in-the-loop pipelines that combine efforts made by humans and machines. We will introduce data labeling via public crowdsourcing marketplaces and present the critical components of efficient data labeling. Then, we will run a practical session, where participants address a challenging real-world Information Retrieval for e-Commerce task, experiment with selecting settings for the labeling process, and launch their label collection project on real crowds within the tutorial session. We will present useful quality control techniques and provide the attendees with an opportunity to discuss their annotation ideas. Methods and techniques described in this tutorial can be applied to any crowdsourced data and are not bound to any specific crowdsourcing platform.

Agenda

09:00 - 09:45

The role of Human-in-the-Loop in building Search engines

09:45 - 10:30

Ranking and Quality Metrics

10:30 - 11:00

Break

11:00 - 11:45

Human-in-the-Loop Essentials

11:45 - 12:30

Hands-On Practice Session

12:30 - 14:00

Lunch Break

14:00 - 14:45

Results aggregation and implementation into the ML pipeline

14:45 - 15:30

Metric computation

15:30 - 16:00

Break

16:00 - 17:30

Results discussion and Conclusion

Speakers

Image
Alexey Drutsa
TolokaDeputy CEO, COOProfile link
Image
Dmitry Ustalov
TolokaHead of ResearchProfile link
Image
Nikita Pavlichenko
TolokaMachine Learning ResearcherProfile link
Image
Daria Baidakova
TolokaDirector of Educational ProgramsProfile link
Image
Boris Tseytlin
TolokaMachine Learning ResearcherProfile link

Don't miss

Be sure to attend our informative workshops,
tutorials, and webinars.

Fractal