In this talk, we report our experience of running behavioral studies on the Toloka crowdsourcing platform.
The COVID-19 pandemic put a severe strain on conventional offline human subjects studies, forcing behavioral labs worldwide to transition to online interactive experiments. The ‘traditional’ platforms used for this purpose, such as Amazon Mechanical Turk and Prolific, were flooded with researchers' requests, resulting in deteriorating data quality. Moreover, these platforms have limited geographical coverage (the US for MTurk and the UK for Prolific), limiting the generality of findings. All these factors created a demand for alternatives, and in this talk, we report our experience of running behavioral studies on the Toloka crowdsourcing platform. We demonstrate how to run both asynchronous studies without real-time interactions between participants and synchronous, interactive studies where the partner’s decisions are delivered to participants in real time. We discuss several major challenges researchers face when conducting interactive studies online, such as controlling for attention and drop-offs.