Behavioral studies on crowdsourcing platforms

Overview

The COVID-19 pandemic put a severe strain on conventional offline human subjects studies, forcing behavioral labs worldwide to transition to online interactive experiments. The ‘traditional’ platforms used for this purpose, such as Amazon Mechanical Turk and Prolific, were flooded with researchers' requests, resulting in deteriorating data quality. Moreover, these platforms have limited geographical coverage (the US for MTurk and the UK for Prolific), limiting the generality of findings. All these factors created a demand for alternatives, and in this talk, we report our experience of running behavioral studies on the Toloka crowdsourcing platform. We demonstrate how to run both asynchronous studies without real-time interactions between participants and synchronous, interactive studies where the partner’s decisions are delivered to participants in real time. We discuss several major challenges researchers face when conducting interactive studies online, such as controlling for attention and drop-offs.

Speaker

Philipp Chapkovski

Higher School of EconomicsPhilipp Chapkovski is a post-doctoral research fellow at the International Laboratory for Experimental and Behavioral Economics at the HSE University, Moscow. He wrote his thesis at the European University Institute in Florence, studying collective sanctions and their effect on cooperation and intergroup conflict using lab and online experiments.