The toloker sees the search query and 9 products from the search results.
Their task is to rate the results, choosing among
- "might fit"
- "doesn't fit"
- "page not found"
The last value helps identify technical problems on the website. To simulate user behavior as accurately as possible, the developers recreated the interface of the online store in an iframe.
At the same time as the task was launched on Toloka, the search queries were labeled using rules. The focus was on popular queries, in order to improve their search results first.
Labeling with rules made it possible to get data faster using a small number of queries, and the results for top queries were good. But there were also disadvantages: ambiguous queries can't be evaluated using rules, and there are many controversial situations. This method also proved rather expensive in the long term.
Manual labeling doesn't have those disadvantages. In Toloka, you can collect the opinions of a large number of tolokers and get more granular evaluations, which lets you analyze search results more deeply. After the initial setup, the platform works stably and processes large amounts of data.