Video classification

Tip.

Run the project in the Sandbox first. This helps you avoid making mistakes and spending money on a task that isn't working right.

Projects of this type are intended for video classification. You can use them for:
  • Content moderation.
  • Sorting video clips by specified categories.
  • Rating video clips based on how much the performers like them.
  • Detecting noise and other visual imperfections in the video.

For example, you have a lot of video clips, and you need to determine which ones are of higher quality. To do this, create a task in which the performers will watch the video clips and evaluate their quality.

  1. Before you start
  2. Create a project
  3. Create a pool
  4. Upload a TSV file with tasks.
  5. Get the results

Before you start

  • To use your video clips in the project, you need to upload them to a storage from which you can get file links. For example, your server, video hosting, or cloud storage:

    • Instructions on how to use files from Yandex.Cloud.
    • If you use video clips from YouTube, replace the https://www.youtube.com/watch?v=example links with https://www.youtube.com/embed/example.
  • Make sure that the video clips look the same in the desktop and mobile versions of Toloka. A video clip may not play or play incorrectly in the mobile version. In this case, when creating a pool, restrict access to the pool for the performers with mobile devices.

    Learn more about configuring tasks for mobile devices.

    Note.

    The video player supports only MP4 format with H.264 or H.265 encoding.

  • If you have a complex project, register in the sandbox and create a project there. There you can:

    1. Test the project settings as a performer.
    2. Then transfer them to the production version.
    3. Launch the project for real performers.

    This helps you avoid making mistakes and spending unnecessary money on a task that doesn't work.

Create a project

Open Toloka for requesters.

  1. Choose a template:

    1. Click Create project.

    2. Select the Video moderation template.
  2. Provide general information:

    1. Enter a clear name and a short description for the project. Performers will see this in the task list.

    2. Optionally add a Private comment.

    3. Click Save.

  3. Edit the task interface in the editor you selected:

    Template Builder
    HTML/CSS/JS editor
    1. The task interface describes how the elements should be arranged in the task.

      Use the ready-made template for this project with pre-configured validation, keyboard shortcuts, and task layout. The performer won't be able to submit a response without selecting one of the options.

      For more information, see the Template Builder Help:

    2. To specify what data you will pass to the performer and receive in response, create input and output data fields.

      What are input and output data?

      Input data is types of objects that are passed to the performer for completing the task. For example, this could be a text, an image, or geographic coordinates.

      Output data is types of objects that you receive after the task is completed. For example, this could be one of several response options, typed text, or an uploaded file.

      Input data fields are created from the code on the Example of input data tab.

      The output data fields depend on the components that use data.output and values supported by it.

      Click Show specifications to see the input and output data fields.

      Learn more about input and output data fields in the Template Builder Help.

      In this project:

      • Input data field: video — A link to a video.

      • Output data field: result — string for saving the performer's response.

    1. The task interface describes how the elements should be arranged in the task.

      In the HTML interface, use the standard HTML tags and special expressions in double curly brackets for input and output data fields.

      For this project, leave the HTML, JS, and CSS blocks unchanged.

      If you use a video from YouTube
      Add the Iframe element to the interface:
      <iframe src={{video}} width="560" height="315" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
      Copied to clipboard
    2. In the Data specification section, you can configure the input and output data fields.

      What are input and output data?

      Input data is types of objects that are passed to the performer for completing the task. For example, this could be a text, an image, or geographic coordinates.

      Output data is types of objects that you receive after the task is completed. For example, this could be one of several response options, typed text, or an uploaded file.

      Learn more about the Specification parameters in Input and output data.

      In this project:

      • Input data field: video — A link to a video.

        Attention. If you use a video from YouTube, set the string data type for the video input field.
      • Output data field: result — string for saving the performer's response.

    3. Click to see the performer's view of the task.

      Note. The project preview shows one task with standard data. You can define the number of tasks to show on the page later.
    4. In the window that opens, check if the task options work correctly. In the lower-right corner, click Submit.

    5. To exit preview mode, click Exit in the lower-left corner.

    6. Save the changes.

    1. Write short and clear instructions. Describe what needs to be done and give examples in them.

      You can prepare instructions in HTML format, then copy and paste into the editor. Click <> to switch to HTML mode.

    2. Click Finish.

Learn more about working with the project in Project.

Create a pool

A pool is a set of paid tasks sent out for completion at the same time.

  1. Open the page of the project that evaluates the video quality.

  2. Click the Add a pool button.

  3. Specify the Pool name.

  4. (optional) Enter a private description.

    1. Turn on the Add a private description option.

    2. In the Private description field, enter the pool description. This information is available only to you.

  5. In the Price per task suite section, find Price in US dollars and specify the price. For example, 0.01.

    What is a task suite?

    A task suite can contain one or several tasks that are shown on the same page. If the tasks are simple, you can add 10-20 tasks per suite. Don't make task suites too long because it slows down loading speed for performers.

    Performers get paid for completing the entire task suite.

    The number of tasks per suite is set when uploading tasks.

    What is a fair price for a task suite?

    The general rule of pricing is the more time the performer spends to complete the task, the higher the price is.

    You can register in Toloka as a performer and find out how much other requesters pay for tasks, or see examples of cost for different types of tasks.

  6. In the Performers section, filter performers.

    • To make the task available only to performers who speak Russian:

      1. Click Add filter.

      2. Find the Audience presets block in the list and choose the Russian speaking users filter.

    • If the task isn't suitable for performers with mobile devices:

      1. Restrict access from the mobile browser. Add the User agent type filter.

      2. Set the following values: User agent type Mobile browser.

      3. Restrict access from smartphones and tablets. Add the Device type filter.

      4. Set the following values: Device type Smartphone or Tablet.

  7. Quality control rules allow you to filter out careless performers. In the Quality control section, add the following quality control blocks:

    • Control tasks — filters out performers who often make mistakes in the control tasks.

      1. Click Add Quality Control Rule.

      2. Find the Rules block in the list and choose Control tasks.

      3. Set a rule for the control task: if the number of responses to the control questions is ≥ 3 and correct responses (%) to the control questions is < 60, then ban the performer from the project for 10 days. Specify Control task as a reason.

        This means that if a performer completed more than three control tasks and gave incorrect answers in more than 60% of them, the performer will be banned and won't be able to complete tasks on this project for 10 days.

    • Fast responses —Filters out performers who respond too quickly.

      1. In the Recent values to use field, enter the number of the performer's recently completed assignments. For example, 5.
      2. In the Minimum time per task suite field, enter the time in seconds. For example, 20.
      3. Set a rule for a fast response: if the number of fast responses ≥ 1, then ban on requester for 10 days. In the Reason field, enter Fast responses.

        This means that a user who completes at least one assignment in less than 20 seconds won't be able to access your tasks for 10 days.

    • Majority vote — a check based on matching responses from the majority of performers.

      1. Set Accept as majority to 2.

      2. Set the rule: if number of responses ≥ 5 and correct responses (%) < 50, then ban the performer on project for 10 days. Specify the reason Doesn't match the majority.

        This means that if a performer completed more than ten tasks and most of their responses don't match the opinion of the majority of performers, they won't be able to complete project tasks for 10 days.

        Note.

        The rule takes effect when the number of responses for the task is equal to the overlap. To reach the required number of responses faster, enable the Keep task order option in the pool settings.

    • Captcha — Prevents robots from completing tasks.

      1. In the Recent values to use field, enter the number of the performer's recently completed assignments. For example, 10.
      2. Set a captcha rule: if the number of responses ≥ 5 and correct responses (%) < 65, then ban on project for 10 days. In the Reason field, enter Captcha.

        This means that if a performer enters captchas correctly in less than 65% of cases, they won't be able to access tasks in the project for 10 days.

    Note.

    You can copy quality control settings from another pool. To do this, click Copy settings from in the Users filter section.

  8. Set overlap, which is the number of performers to complete the same task. In the Overlap section, specify the Overlap field value. For tasks of this type it is usually 3-5.

  9. (optional) In the Speed/quality ratio section, specify the percentage of top-rated performers who will get access to the pool. Learn more in the Speed/quality balance section.

  10. Adjust the pool parameters:

    1. Specify the Time for completing a task suite in seconds field value. There should be enough time to read the instructions, load the task, watch the video, and respond. For example, 1200 seconds.
    2. Turn on the Keep task order option.
  11. Click Save.

Upload a TSV file with tasks.

Download the sample upload file. You can find it on the pool page. At the top-left of the page, there are links to TSV files with regular, control, and training tasks. Use it to prepare your own file with tasks.

  1. On the pool page, click Upload. In the window that opens, you can also download a sample TSV file by clicking Sample file for uploading tasks.

  2. In the window that opens, configure the file upload settings:

    1. Choose Smart mixing.

    2. In the Main tasks field, specify 9.

    3. In the Training tasks field, specify 0.

    4. In the Control tasks field, specify 1.

    5. Click Upload.

    6. In the window that opens, choose the TSV file with tasks to upload and click Open.

    7. In the window that opens, check the number of tasks and click Add.

  3. Create a control task.

    1. Click Edit.

      Note. If you selected something else instead of smart mixing, click Edit. If this button is missing, delete the file and upload it again.
    2. In the window that opens, click Create control tasks.

    3. In the window that opens, in the Create control task section, mark the result item on the left.

    4. Choose the correct answer to the question.

    5. Click Save and go to next.

    6. Click Rate the video quality to exit task markup mode.

    Note.

    In small pools, control tasks should be 10% of all tasks. Include different versions of correct responses in equal amounts. View the distribution of responses on the Edit tasks page, Control tasks tab.

  4. Click to start the pool.

Get the results

  1. Next to the Download results button, click .

  2. Choose Dawid-Skene aggregation model from the list. Learn more about Result aggregation based on the Dawid-Skene model.

  3. In the window that opens, click Yes.

  4. At the top of the page, click View the list of operations.

  5. When the operation is complete, download the TSV file with the results. To do this, click Download in the Files column.

Tip.

To avoid unwanted errors, we recommend that you first run your project in the Sandbox.

Troubleshooting