WSDM 2023 Crowd Science Workshop
CANDLE: Collaboration of Humans and Learning Algorithms for Data Labeling.
Overview
Crowdsourcing has been used to produce impactful and large-scale datasets for Machine Learning and Artificial Intelligence (AI), such as ImageNET, SuperGLUE, etc. Since the rise of crowdsourcing in the early 2000s, the AI community has been studying its computational, system design, and data-centric aspects at various angles at such workshops as CSS, CrowdML, DCAI, and HILL.
We welcome studies on developing and enhancing crowdworker-centric tools that offer task matching, requester assessment, and instruction validation, among other topics. We are also interested in exploring methods that leverage crowdworkers as a resource for improving the recognition and performance of machine learning models. Thus, we invite studies of active learning techniques, methods for joint learning from noisy data and from crowds, novel approaches for crowd-computer interaction, repetitive task automation, and role separation between humans and machines. Moreover, we invite works on designing and applying such techniques in various domains, including e-commerce and medicine.
Agenda
* The time is indicated in Singapore time zone (UTC+08)
Opening
Invited talk "Human Input is Indispensable in the Age of Generative AI" by Ujwal Gadiraju
Paper oral "AI Decision Systems with Feedback Loop Active Learner" by Mert Kosan et al.
Paper oral "Active Learning via Density-based Space Transformation" by Mohammadhossein Bateni et al.
Coffee break
Invited talk "Machine-in-the-loop: A New Paradigm of Crowdsourcing for Wikipedia Editing" by Djellel Difallah
Paper oral "To Aggregate or Not? Learning with Separate Noisy Labels" by Jiaheng Wei et al.
Paper oral "The determination of the learning performance based on assessment item analysis" by Doru Anastasiu Popescu et al.
Paper oral "Utilising crowdsourcing to assess the effectiveness of item-based explanations of merchant recommendations" by Oleg Lashinin et al.
Closing remarks
Speakers
Organizers
Don't miss out
tutorials, and webinars.
