Using A/B experiments

A/B testing is a method of user experience research. It includes statistical testing of hypotheses and choosing the most appropriate option from multiple ones.

When to use

Use A/B experiments to:

  • Find out how changes in the instructions, interface, filters, or in the quality control rules affect the Tolokers responses.

  • Launch the pools or the projects with different settings for independent groups of Tolokers.

  • Control the labeling quality. If your pool is small, it's hard to track how Toloker's labeling quality changes over time. For example, the Toloker had passed the exam but then for some reason their performance got worse. The experiments allow you to filter out such Tolokers even if your pool has few tasks.

How it works

  • Each Toloker has an id number ranging from 1 to 100 (100 independent groups of Tolokers). Parameter is set to a Toloker like a skill and Tolokers will always be placed into the the same group.

  • Use the Experiment group (1-100) filter in the pool settings to select Tolokers from one or several groups.

  • To create an A/B experiment add two pools with different values of the Experiment group (1-100) filter.

Alert

Use the Experiment group (1-100) filter carefully. If you set the Experiment group (1-100) filter = 1 in one pool and Experiment group (1-100) filter = 2 in another pool, then about 98% of the Tolokers will not see your tasks.

Troubleshooting

Can I select two filters at once?
Can I run the experiments within one project?
Can part of the Tolokers do tasks from both pools?
Contact support

Last updated: February 15, 2023

Introduction
Getting started
Important tips
Useful recommendations
Working with Toloka
Projects
Pools
Tasks
Results
Project analysis
Toloka settings
Task interface
Template Builder
HTML/CSS/JS editor
Help and support
FAQTroubleshootingSupportGlossary