WSDM 2023 Crowd Science Workshop

CANDLE: Collaboration of Humans and Learning Algorithms for Data Labeling.



Crowdsourcing has been used to produce impactful and large-scale datasets for Machine Learning and Artificial Intelligence (AI), such as ImageNETSuperGLUE, etc. Since the rise of crowdsourcing in the early 2000s, the AI community has been studying its computational, system design, and data-centric aspects at various angles at such workshops as CSS, CrowdMLDCAI, and HILL.

We welcome studies on developing and enhancing crowdworker-centric tools that offer task matching, requester assessment, and instruction validation, among other topics. We are also interested in exploring methods that leverage crowdworkers as a resource for improving the recognition and performance of machine learning models. Thus, we invite studies of active learning techniques, methods for joint learning from noisy data and from crowds, novel approaches for crowd-computer interaction, repetitive task automation, and role separation between humans and machines. Moreover, we invite works on designing and applying such techniques in various domains, including e-commerce and medicine.


* The time is indicated in Singapore time zone (UTC+08)

1:30 pm - 1:35 pm


1:35 pm - 2:20 pm

Invited talk "Human Input is Indispensable in the Age of Generative AI" by Ujwal Gadiraju

2:20 pm - 2:40 pm

Paper oral "AI Decision Systems with Feedback Loop Active Learner" by Mert Kosan et al.

2:40 pm - 3:00 pm

Paper oral "Active Learning via Density-based Space Transformation" by Mohammadhossein Bateni et al.

3:00 pm - 3:30 pm

Coffee break

3:30 pm - 4:15 pm

Invited talk "Machine-in-the-loop: A New Paradigm of Crowdsourcing for Wikipedia Editing" by Djellel Difallah

4:15 pm - 4:30 pm

Paper oral "To Aggregate or Not? Learning with Separate Noisy Labels" by Jiaheng Wei et al.

4:30 pm - 4:45 pm

Paper oral "The determination of the learning performance based on assessment item analysis" by Doru Anastasiu Popescu et al.

4:45 pm - 5:00 pm

Paper oral "Utilising crowdsourcing to assess the effectiveness of item-based explanations of merchant recommendations" by Oleg Lashinin et al.

5:00pm - 5:05 pm

Closing remarks


Ujwal Gadiraju
Delft University of TechnologyAssistant ProfessorProfile link
Mert Kosan
UC Santa BarbaraResearch AssistantProfile link
Djellel Difallah
New York University Abu DhabiAssistant Professor of Computer ScienceProfile link
Jiaheng Wei
University of California, Santa CruzResearch AssistantProfile link


Dmitry Ustalov
TolokaHead of ResearchProfile link
Alisa Smirnova
TolokaResearcherProfile link
Saiph Savage
Northeastern UniversityAssistant ProfessorProfile link
Niels van Berkel
Aalborg UniversityAssociate ProfessorProfile link
Yang Liu
University of California, Santa CruzAssistant ProfessorProfile link

Don't miss out

Be the first to hear about our workshops, 
tutorials, and webinars.