Problem: All people have biases, whether they are aware of them or not. Importantly, manifestations of these biases are often too subtle to be detected, but can hurt the quality of collected data. For example, performers can be influenced by certain aspects of the task design without realizing it — missing information that causes doubt, preference for options that are given first, similarity between response options — and these issues can decrease the quality of collected data.
Solution: Crowdsourcing projects can take a practical approach to eliminating the opportunity for bias. Careful instructions can reduce uncertainty and simplify the task for performers. To avoid social influence, performers are usually not allowed to see answers given by other people. Another common strategy is to use a randomized interface that shuffles the order of response options for each performer.