YOUR FEEDBACK IS CRUCIAL
Fill out a brief survey to help us improve
our Knowledge Base.
Intent classification — Step-by-step instructions

The challenge

We need to define which class the search query belongs to and distribute the queries between several categories inside the class. There’s a list of queries (related to travel and dining), each with an unknown class and category. Performers are asked to first select the search query’s class and then define the category it belongs to within this class. Here’s what it might look like:

Video: 7 basic steps to run a project
Create
a project
Create
a training pool
Create
a task pool
Upload a file
with data
Create
control tasks
Launch
the pool
Get the
results

Create a project

Interface code

{
  "view": {
    "type": "view.list",
    "items": [
      {
        "type": "view.alert",
        "theme": "info",
        "label": "QUERY",
        "content": {
          "type": "view.text",
          "content": {
            "type": "data.input",
            "path": "query"
          }
        }
      },
      {
        "type": "field.button-radio-group",
        "label": "Please select query category:",
        "options": [
          {
            "label": "Kitchen",
            "value": "kitchen"
          },
          {
            "label": "Travel",
            "value": "travel"
          }
        ],
        "data": {
          "type": "data.output",
          "path": "domain"
        },
        "validation": {
          "type": "condition.required"
        }
      },
      {
        "type": "helper.if",
        "condition": {
          "type": "condition.equals",
          "data": {
            "type": "data.output",
            "path": "domain"
          },
          "to": "kitchen"
        },
        "then": {
          "type": "field.radio-group",
          "label": "What is the intent of the query?",
          "options": [
            {
              "label": "Restaurant review",
              "value": "review",
              "hint": "Anything related to whether a restaurant is worth the visit"
            },
            {
              "label": "Restaurant reservation",
              "value": "reservation",
              "hint": "Intention to reserve a table in the restaurant"
            },
            {
              "label": "Nutritional information",
              "value": "nutrition",
              "hint": "Intention to know information about how healthy food or amount of calories "
            },
            {
              "label": "Recipe",
              "value": "recipe",
              "hint": "Anything related to how to prepare food or food substitution"
            }
          ],
          "data": {
            "type": "data.output",
            "path": "intent"
          },
          "validation": {
            "type": "condition.required",
            "hint": "please select the relevant intent"
          }
        }
      },
      {
        "type": "helper.if",
        "condition": {
          "type": "condition.equals",
          "data": {
            "type": "data.output",
            "path": "domain"
          },
          "to": "travel"
        },
        "then": {
          "type": "field.radio-group",
          "label": "What is the intent of the query?",
          "options": [
            {
              "label": "Book a hotel",
              "value": "bookhotel",
              "hint": "Anything related to reservation a flight or hotel"
            },
            {
              "label": "Timezone",
              "value": "timezone",
              "hint": "Intention to get information about timezone"
            },
            {
              "label": "Travel suggestion",
              "value": "travel_suggestion",
              "hint": "Query related to recommendation on a travel"
            },
            {
              "label": "Exchange rate",
              "value": "exchange_rate",
              "hint": "Intention to know price to buy such currency"
            },
            {
              "label": "Vaccine",
              "value": "vaccine",
              "hint": "Query which contains information about vaccines in certain region"
            }
          ],
          "data": {
            "type": "data.output",
            "path": "intent"
          },
          "validation": {
            "type": "condition.required",
            "hint": "please select the relevant intent"
          }
        }
      }
    ]
  },
  "plugins": [
    {
      "type": "plugin.toloka",
      "layout": {
        "kind": "scroll",
        "taskWidth": 300
      }
    }
  ]
}

Create a training pool

Training dataset
Prepare a training dataset as shown in our example.
Origin
@inproceedings{larson-etal-2019-evaluation,
title = "An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction",
author = "Larson, Stefan and Mahendran, Anish and Peper, Joseph J. and Clarke, Christopher and Lee, Andrew and Hill, Parker and Kummerfeld, Jonathan K. and Leach, Kevin and Laurenzano, Michael A. and Tang, Lingjia and Mars, Jason",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)",
year = "2019",
url = "https://www.aclweb.org/anthology/D19-1131" }
License: CC BY 3.0

Create a task pool

Upload a file with data

Dataset
Prepare a TSV file with tasks as shown in our example.
You can also upload a file that already contains golden sets: example.
Origin
@inproceedings{larson-etal-2019-evaluation,
title = "An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction",
author = "Larson, Stefan and Mahendran, Anish and Peper, Joseph J. and Clarke, Christopher and Lee, Andrew and Hill, Parker and Kummerfeld, Jonathan K. and Leach, Kevin and Laurenzano, Michael A. and Tang, Lingjia and Mars, Jason",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)",
year = "2019",
url = "https://www.aclweb.org/anthology/D19-1131" }
License: CC BY 3.0

Create control tasks

Launch the pool

Get the results

Application for corporate training
We are offering corporate training to help you solve existing challenges and develop
an internal team of Crowd Science Architects (CSA).
Mon Oct 04 2021 12:19:30 GMT+0300 (Moscow Standard Time)