Using A/B experiments
A/B testing is a method of user experience research. It includes statistical testing of hypotheses and choosing the most appropriate option from multiple ones.
When to use
Use A/B experiments to:
Find out how changes in the instructions, interface, filters, or in the quality control rules affect the Tolokers responses.
Launch the pools or the projects with different settings for independent groups of Tolokers.
Control the labeling quality. If your pool is small, it's hard to track how Toloker's labeling quality changes over time. For example, the Toloker had passed the exam but then for some reason their performance got worse. The experiments allow you to filter out such Tolokers even if your pool has few tasks.
How it works
Each Toloker has an
idnumber ranging from 1 to 100 (100 independent groups of Tolokers). Parameter is set to a Toloker like a skill and Tolokers will always be placed into the the same group.Use the Experiment group (1-100) filter in the pool settings to select Tolokers from one or several groups.
To create an A/B experiment add two pools with different values of the Experiment group (1-100) filter.
Use the Experiment group (1-100) filter carefully. If you set the Experiment group (1-100) filter = 1 in one pool and Experiment group (1-100) filter = 2 in another pool, then about 98% of the Tolokers will not see your tasks.