Search relevance evaluation— Step-by-step instructions

The challenge

We have a set of search queries and products on a website. We need to determine the extent to which each query is relevant to the corresponding product on the website. We ask performers to look at the search query and the product image from the website and rate the relevance level.

Video: 8 basic steps to run a project
Create
a project
Create
a training pool
Create
an exam pool
Create
a task pool
Upload a file
with data
Create
control tasks
Launch
the pool
Get the
results

Create a project

Interface code

{
  "view": {
    "type": "view.list",
    "items": [
      {
        "type": "layout.sidebar",
        "minWidth": 400,
        "content": {
          "type": "view.list",
          "size": "m",
          "direction": "vertical",
          "items": [
            {
              "type": "view.image",
              "maxWidth": 350,
              "url": {
                "type": "data.input",
                "path": "imagepath"
              }
            },
            {
              "type": "view.markdown",
              "label": "Product title:",
              "content": {
                "type": "data.input",
                "path": "title"
              }
            }
          ]
        },
        "controls": {
          "type": "view.list",
          "direction": "vertical",
          "items": [
            {
              "type": "view.alert",
              "theme": "info",
              "label": "Search query",
              "content": {
                "type": "view.text",
                "content": {
                  "type": "data.input",
                  "path": "query"
                }
              }
            },
            {
              "type": "view.action-button",
              "label": "Search query in Google",
              "action": {
                "type": "action.open-link",
                "payload": {
                  "type": "data.input",
                  "path": "search_url"
                }
              }
            },
            {
              "type": "view.divider"
            },
            {
              "type": "field.radio-group",
              "label": "Choose relevance class",
              "options": [
                {
                  "label": "Relevant",
                  "value": "relevant"
                },
                {
                  "label": "Slightly relevant",
                  "value": "relevant_minus"
                },
                {
                  "label": "Irrelevant",
                  "value": "irrelevant"
                }
              ],
              "data": {
                "type": "data.output",
                "path": "result_class"
              },
              "validation": {
                "type": "condition.required"
              }
            }
          ]
        }
      }
    ]
  },
  "plugins": [
    {
      "type": "plugin.toloka",
      "layout": {
        "kind": "scroll",
        "taskWidth": 600
      }
    }
  ]
}

Create a training pool

Training dataset
Prepare a TSV file with tasks as shown in our example.
Origin
License: PDDL

Create an exam pool

Exam dataset
Prepare a TSV file with tasks as shown in our example.
Origin
License: PDDL

Create a task pool

Upload a file with data

Dataset
Prepare a TSV file with tasks as shown in our example.
You can also upload a file that already contains golden sets: example.
Origin
License: PDDL

Create control tasks

Dataset with control tasks
You can upload a file that already contains golden sets: example.
Origin
License: PDDL

Launch the pool

Get the results

Application for corporate training
We are offering corporate training to help you solve existing challenges and develop
an internal team of Crowd Science Architects (CSA).
Thu Oct 07 2021 11:17:00 GMT+0300 (Moscow Standard Time)