This ICML 2021 workshop by Toloka aims to provide a comprehensive picture of how crowdsourcing can be applied to real life AI production.
AI development today rests on three pillars: algorithms, hardware, and data. Ironically, the further AI moves towards new application areas, the more it depends on human efforts: more and more often data for training and validating AI models cannot be collected in any other way than by humans.
AI solutions require data for training and validating models that are not only high-quality and scalable to support growing industry needs but also flexible enough to support a large variety of use cases and data collection scenarios.
Toloka's mission is to create an environment for AI data production that is fully aligned with industry needs: quality, scalability, flexibility.As a result, Toloka is a multifaceted solution with:
The Toloka workshop aims to cover these aspects and provide a comprehensive picture of how crowdsourcing can be applied to real life AI production.
The workshop will feature:Keynotes:
Demo: Automated Pipeline for E-Commerce Item Retrieval and Ranking
Dmitry Ustalov, Vladimir Losev, and Oleg Pavlov will provide a hands-on demonstration of how crowdsourcing can help address an e-commerce item retrieval and ranking task. In particular, they will show the attendees how to build a human-in-the-loop pipeline that combines both crowdsourced data and ML models to obtain a reliable ground-truth dataset on the Toloka platform.
The Toloka team will demonstrate how interdependent data labeling processes can be programmatically combined using the Toloka-Kit Python library, and how the final annotation results can be obtained using the Crowd-Kit computational quality control library.
Evolution of data production paradigm in AI
The Practice of Crowdsourcing
Data Annotation at Scale: a Core Expertise of Modern ML
The Future of Work for Performers: Empowering the People behind AI
Automated Pipeline for E-Commerce Item Retrieval and Ranking
Demo by Dmitry Ustalov, Vladimir Losev, Oleg Pavlov
Retail Week Live 2023
Next-level ecommerce: A winning formula to surpass your competitors
Data Council Austin 2023
How to ensure your model does not drift? From Human-in-the-Loop concept to building fully adaptive ML models using crowdsourcing