Toloka documentation

Video classification

In this tutorial, you will learn how to run video classification in Toloka. We will use a project preset designed specifically for this type of data labeling.

Video classification is a type of data labeling task with a video player and several response options. A Toloker watches the video clip and chooses one of the given response options.

You can use this type of project for:

  • Content moderation.

  • Sorting video clips by specified categories.

  • Rating video clips based on how much the Tolokers like them.

  • Detecting noise and other visual imperfections in the video.

Prerequisites

Before you begin:

  • Make sure you are registered in Toloka as a requester.

  • Top up your Toloka account. If you are unsure about the budget, you can do that later in this tutorial. Toloka will display the budget estimate for your project.

  • Upload your video clips to a storage from which you can get file links. For example, your own server, a video hosting, or a cloud storage.

    Restriction

    The video player supports only MP4 format with H.264 or H.265 encoding.

  • Make sure that the video clips look the same in the desktop and mobile versions of Toloka. A video clip may not play or play incorrectly in the mobile version. In this case, when creating a pool, you will need to restrict access to the pool for the Tolokers with mobile devices.

    Learn more about configuring tasks for mobile devices.

Choose a preset

We recommend starting with a project preset for easier configuration and better results.

  1. Follow this link, or create a project manually:

    1. In the main menu, choose the Projects tab, and click Create a project.

      Choose a preset. Step 1

    2. Select the Hand gesture classification preset.

  2. Click Choose this preset in the pop-up tab.

Create a project

Set up how your tasks will look for Tolokers. Tolokers are people around the world who get paid for completing your tasks.

  1. Under General information, add the project name and description.

    • Name to show Tolokers: In 2–5 words, state the general idea of the project.

    • Description for Tolokers: In a couple of sentences, explain what you expect Tolokers to do. This is just an overview. You will write instructions later.

    Create a project. Step 1

  2. In the Task interface section, set up what your tasks will look like. This preset has a task template with validation, keyboard shortcuts, and task layout pre-configured.

    Note

    This tutorial uses Template Builder, but you can use the HTML/JS/CSS editor for the same purpose.

    1. Using the Visual editor, add your data to the Config section:

      • Question Tolokers will see in your task: Write a question that matches the responses Tolokers need to choose from. All tasks in a project use the same question.

      • Paste link to a sample video: This video is only used to display the task interface preview on the right.

    2. Set answer options is pre-filled with sample answers. Replace the samples with your categories. Note that Other and Error are separate entities.

      • Answer option for Tolokers: This is the label that Tolokers will see. Make sure it is clear and correct.

      • Name in labeling results: This is the value you will see in the file with the labeling results.

    3. If you use videos from YouTube

      To display videos from YouTube, you should transform links to them. For example, the https://www.youtube.com/watch?v=your_video_example link should be replaced with https://www.youtube.com/embed/your_video_example.

      1. In the Paste link to a sample video field, specify the https://www.youtube.com/embed/your_video_example link. You can use a link to any of your videos. It is only used to display the task interface preview on the right.

        Create a project. Step 2.3.1

      2. Go to the Code editor tab. In the Config section, replace the view.video value with view.iframe.

        Create a project. Step 2.3.2

        Note

        To learn about other properties of the Config section, their possible values and the impact on the task interface, see the Template Builder Help.

    4. Raw task data is stored in the XSLX, TSV, or JSON format. The labeling results are presented in a TSV file. The Data specification section determines which parameters these files might contain.

      Click Show specifications and check the values:

      • Input data: Parameters in the file with raw task data.

      • Output data: Parameters in the file with labeling results.

      Input data and Output data match the task interface you set up in Template Builder. Check that there are fields for all data types you use for your tasks, and for the ones you want to see in the results file.

  3. Under Instructions for Tolokers, add the instructions Tolokers will see when they start doing your tasks. You can add text, tables, and images to your instructions.

    Check the sample text of the instructions, and update it to fit your project.

    Tip

    When writing instructions, remember that most Tolokers don’t know anything about your tasks beforehand. Make sure your instructions are as clear as possible, but not too wordy. For successful data labeling, try to strike a balance between covering all the essentials and keeping it short. Learn more in our knowledge base.

  4. To save your data and continue, click Create a project.

    Create a project. Step 4

Create a pool

A pool is a set of tasks sent out to Tolokers at the same time. One project can have many pools. When creating a pool, you set up pricing, audience filters for Tolokers, and quality control.

  1. Click Create new pool.

  2. Under General information, set the Pool name.

  3. Under Audience, set up filters to select Tolokers for your pool.

    1. Clear My tasks may contain shocking or pornographic content if your project has none of those.

    2. To select Tolokers based on their language, location, age, gender, and other parameters, click the Add filter button.

      Create a pool. Step 3.2

    3. Tasks in pools will automatically be available in the web version of Toloka and the mobile app. If you want to change the default settings and limit the visibility of the task for any of the versions, add the Client filter and select the desired value: Toloka web version or Toloka for mobile.

    4. Use the Speed/quality balance slider to change the number of Tolokers who can see your tasks. Move the slider to the right to exclude Tolokers with lower ratings from participating in your project.

      Create a pool. Step 3.3

  4. Under Quality control, set quality control rules for more accurate results:

    1. Keep the preconfigured quality control rules as is:

      • Fast responses: This rule filters out Tolokers who complete tasks too fast. The default settings mean that Tolokers are banned from the project for 1 day if they respond to 4 out of 5 tasks in less than 15 seconds.

      • Majority vote: This rule accepts the most popular response as the correct one, and allows you to filter out Tolokers who answer incorrectly. The default settings mean that Tolokers who give correct responses to less than 40% of tasks are banned from the project for 1 day. Accept as majority set to 2 means that 2 similar responses out of all responses given to a single task will be considered as the correct answer.

    2. To filter out Tolokers who often make mistakes in the control tasks, click Add a quality control rule → Control tasks. Enter the following values:

      Create a pool. Control rule

      This means that if a Toloker completed more than three control tasks and gave incorrect answers in more than 60% of them, they will be blocked and won't be able to complete tasks on this project for 10 days.

      The rule will work if you specify the correct answers for the control tasks. You will do that later in this tutorial.

  5. In Price, set up how much a single task will cost for you.

    1. In Price per task suite, set the amount of money to pay per task suite done by one Toloker. A task suite is a page with a number of tasks. It can contain one or several tasks. If the tasks are simple, you can add 10–20 tasks per suite.

    2. In the Overlap field, define how many Tolokers must do each task.

      The default value (3) means that each task will have 3 responses.

  6. Click Show advanced settings. In Additional settings, specify the Time per task suite, sec.

    This time should be enough to read the instructions, load the task, watch the video clips, and respond (for example, 1,200 seconds).

  7. To save the settings and continue, click Create pool.

Upload data

At this step, upload your task data to Toloka.

  1. Click Upload data.

  2. Attach a prepared dataset or media files.

    1. To download a template, click one of the buttons:

      • Template in XLSX
      • Template in TSV
      • Template in JSON

      For this type of project, the file with tasks must have one parameter. Its name equals INPUT:video, and the values are links to the video clips.

      INPUT:video
      https://yang.s3.yandex.net/finger gestures/20210723_113648.mp4
      https://yang.s3.yandex.net/finger gestures/20210723_113736.mp4
      https://yang.s3.yandex.net/finger gestures/20210723_113521.mp4
      
    2. Open the downloaded file, and replace the sample links with links to your video clips.

    3. Click Select prepared dataset, and upload the file you’ve just made.

    Upload media files from your device to avoid difficulties with placing links in the file with tasks.

    To the right of the Select prepared dataset link, click Select media files and choose files.

    Features:

    • This option is available only for the projects that have a single url type input field in the specification.

    • After uploading, tasks are automatically created with links to the uploaded files.

    Note

    Supported file formats:

    • Images — GIF, JPG, JPEG, PNG, WebP.
    • Audio — FLAC, MP3, WAV, M4A.
    • Video — MP4.
  3. Click Continue.

  4. Tasks are shown to Tolokers in suites. A suite is a single page with multiple tasks. Define how many tasks to include per suite:

    • General tasks: These are tasks for Tolokers to label.

    • Control tasks: These are tasks with predefined answers used to control the quality of responses. You will create them in the next step.

    • Training tasks: These are tasks with predefined answers and explanations for Tolokers. Normally you use training tasks in separate training pools. You don’t have to include them.

    For example, you can add 9 general tasks and 1 control task per suite:

    Upload data. Smart mixing

  5. Click Combine tasks into suites.

  6. To create control tasks, add correct answers to some of your tasks.

    1. Click Edit.

      Upload data. Edit control tasks

    2. On the Edit tasks page, click Create control tasks.

      Upload data. Create control tasks

    3. Check the result checkbox, and select the correct answer for a task. Then, click the Save and go to next button. Add several control tasks this way.

      Upload data. Create control tasks

      Tip

      For large pools (over 1000 tasks), we recommend adding at least 1% of control tasks to the pool. For small pools (around 100 tasks), you need 10% control tasks.

    4. Note the Distribution of correct responses for control tasks graph on the right side of the page. It shows how many control tasks of each type you have. We recommend adding an equal quantity of each correct response.

      Upload data. Distribution of correct responses for control tasks

    5. When you are done adding control tasks, click the pool name in the menu.

      Upload data. Create control tasks

Start labeling

  1. Make sure you have topped up your account.

  2. To send the tasks to Tolokers and begin the labeling process, click Start labeling.

    Start labeling. Step 2

  3. In the pop-up panel, review the budget and click Launch.

See the results

  1. You can see the labeling progress on the pool page. Wait until the labeling is completed.

    See results. Step 1

  2. When the labeling is complete, click the arrow next to the Download results button and choose Run Dawid-Skene model from the drop-down menu. Click Yes in the pop-up window.

    See results. Step 2

  3. Open the same drop-down menu again, and click View aggregations list.

  4. Wait until the aggregation is complete, and click Download. You will get the TSV file with the labeling results:

    • INPUT: The data you uploaded for labeling.

    • OUTPUT: The results of labeling (answers given by Tolokers).

    • CONFIDENCE: The response significance according to the Dawid-Skene model.

See also

Datasets and reference