Knowledge base

Intent classification — Step-by-step instructions

The challenge

We need to define which class the search query belongs to and distribute the queries between several categories inside the class. There’s a list of queries (related to travel and dining), each with an unknown class and category. Performers are asked to first select the search query’s class and then define the category it belongs to within this class. Here’s what it might look like:

Video: 7 basic steps to run a project

Create
a project

Create
a training pool

Create
a task pool

Upload a file
with data

Create
control tasks

Launch
the pool

Get the
results

Download PDF guide

Open in GitHub

Open in Colab

Create a project

Interface code


{
  "view": {
    "type": "view.list",
    "items": [
      {
        "type": "view.alert",
        "theme": "info",
        "label": "QUERY",
        "content": {
          "type": "view.text",
          "content": {
            "type": "data.input",
            "path": "query"
          }
        }
      },
      {
        "type": "field.button-radio-group",
        "label": "Please select query category:",
        "options": [
          {
            "label": "Kitchen",
            "value": "kitchen"
          },
          {
            "label": "Travel",
            "value": "travel"
          }
        ],
        "data": {
          "type": "data.output",
          "path": "domain"
        },
        "validation": {
          "type": "condition.required"
        }
      },
      {
        "type": "helper.if",
        "condition": {
          "type": "condition.equals",
          "data": {
            "type": "data.output",
            "path": "domain"
          },
          "to": "kitchen"
        },
        "then": {
          "type": "field.radio-group",
          "label": "What is the intent of the query?",
          "options": [
            {
              "label": "Restaurant review",
              "value": "review",
              "hint": "Anything related to whether a restaurant is worth the visit"
            },
            {
              "label": "Restaurant reservation",
              "value": "reservation",
              "hint": "Intention to reserve a table in the restaurant"
            },
            {
              "label": "Nutritional information",
              "value": "nutrition",
              "hint": "Intention to know information about how healthy food or amount of calories "
            },
            {
              "label": "Recipe",
              "value": "recipe",
              "hint": "Anything related to how to prepare food or food substitution"
            }
          ],
          "data": {
            "type": "data.output",
            "path": "intent"
          },
          "validation": {
            "type": "condition.required",
            "hint": "please select the relevant intent"
          }
        }
      },
      {
        "type": "helper.if",
        "condition": {
          "type": "condition.equals",
          "data": {
            "type": "data.output",
            "path": "domain"
          },
          "to": "travel"
        },
        "then": {
          "type": "field.radio-group",
          "label": "What is the intent of the query?",
          "options": [
            {
              "label": "Book a hotel",
              "value": "bookhotel",
              "hint": "Anything related to reservation a flight or hotel"
            },
            {
              "label": "Timezone",
              "value": "timezone",
              "hint": "Intention to get information about timezone"
            },
            {
              "label": "Travel suggestion",
              "value": "travel_suggestion",
              "hint": "Query related to recommendation on a travel"
            },
            {
              "label": "Exchange rate",
              "value": "exchange_rate",
              "hint": "Intention to know price to buy such currency"
            },
            {
              "label": "Vaccine",
              "value": "vaccine",
              "hint": "Query which contains information about vaccines in certain region"
            }
          ],
          "data": {
            "type": "data.output",
            "path": "intent"
          },
          "validation": {
            "type": "condition.required",
            "hint": "please select the relevant intent"
          }
        }
      }
    ]
  },
  "plugins": [
    {
      "type": "plugin.toloka",
      "layout": {
        "kind": "scroll",
        "taskWidth": 300
      }
    }
  ]
}

Create a training pool

Training dataset

Prepare a training dataset as shown in our example.
Origin
@inproceedings{larson-etal-2019-evaluation,
title = "An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction",
author = "Larson, Stefan and Mahendran, Anish and Peper, Joseph J. and Clarke, Christopher and Lee, Andrew and Hill, Parker and Kummerfeld, Jonathan K. and Leach, Kevin and Laurenzano, Michael A. and Tang, Lingjia and Mars, Jason",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)",
year = "2019",
url = "https://www.aclweb.org/anthology/D19-1131" }
License: CC BY 3.0

Create a task pool

Upload a file with data

Dataset

Prepare a TSV file with tasks as shown in our example.
You can also upload a file that already contains golden sets: example.
Origin
@inproceedings{larson-etal-2019-evaluation,
title = "An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction",
author = "Larson, Stefan and Mahendran, Anish and Peper, Joseph J. and Clarke, Christopher and Lee, Andrew and Hill, Parker and Kummerfeld, Jonathan K. and Leach, Kevin and Laurenzano, Michael A. and Tang, Lingjia and Mars, Jason",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)",
year = "2019",
url = "https://www.aclweb.org/anthology/D19-1131" }
License: CC BY 3.0

Create a project

Create a training pool

Create a task pool

Upload a file with data

Create control tasks

Launch the pool

Get the results