Voice recording

In this tutorial, you will learn how to run voice recording in Toloka. We will use a project preset designed specifically for this type of data labeling.

Voice recording is a type of data labeling task with the text block to read and the voice recorder button. Using the Toloka mobile app, Tolokers should tap the button and read the text aloud. After getting the results, you can listen to the recordings and download them.

Prerequisites

Before you begin:

• Make sure you are registered in Toloka as a requester.

• Top up your Toloka account. If you are unsure about the budget, you can do that later in this tutorial. Toloka will display the budget estimate for your project.

Choose a preset

We recommend starting with a project preset for easier configuration and better results.

1. Follow this link, or create a project manually:

   1. In the main menu, choose the Projects tab, and click Create a project.

      

   2. Select the Voice recording preset.

2. Click Choose this preset in the pop-up tab.

Create a project

Set up how your tasks will look for Tolokers. Tolokers are people around the world who get paid for completing your tasks.

1. Under General information, add the project name and description.

   • Name to show Tolokers: In 2–5 words, state the general idea of the project.

   • Description for Tolokers: In a couple of sentences, explain what you expect Tolokers to do. This is just an overview. You will write instructions later.

   

2. In the Task interface section, set up what your tasks will look like. This preset has a task template with layout and validation pre-configured.

   Note

   This tutorial uses Template Builder, but you can use the HTML/JS/CSS editor for the same purpose.

   1. In the Config section, you can edit the code to change the appearance and layout of the task elements. For a trial pool, keep the code as is.

      To learn about the properties of the Config section, their possible values and the impact on the task interface, see the Template Builder Help.

   2. In the Input data example section, you can change the sample text. This text is only used to display the task interface preview on the right.

      

   3. Raw task data is stored in the XSLX, TSV, or JSON format. The labeling results are presented in a TSV file. The Data specification section determines which parameters these files might contain.

      Click Show specifications and check the values:

      • Input data: Parameters in the file with raw task data.

      • Output data: Parameters in the file with labeling results.

      Input data and Output data match the task interface you set up in Template Builder. Check that there are fields for all data types you use for your tasks, and for the ones you want to see in the results file.

3. Under Instructions for Tolokers, add the instructions Tolokers will see when they start doing your tasks. You can add text, tables, and images to your instructions.

   Check the sample text of the instructions, and update it to fit your project.

   Tip

   When writing instructions, remember that most Tolokers don’t know anything about your tasks beforehand. Make sure your instructions are as clear as possible, but not too wordy. For successful data labeling, try to strike a balance between covering all the essentials and keeping it short. Learn more in our knowledge base.

4. To save your data and continue, click Create a project.

   

Create a pool

A pool is a set of tasks sent out to Tolokers at the same time. One project can have many pools. When creating a pool, you set up pricing, audience filters for Tolokers, and quality control.

1. Click Add a pool on the project page.

2. Set the Pool name (visible only to you) field. Only you will see this pool name on the project page.

3. Specify the pool description which will be displayed instead of the project description in the task list for Tolokers. By default, Tolokers see the description from the project settings. To use a different description, uncheck the Use project description box and set Public description. If necessary, click + Private comment to add a private project description that only you will see.

4. Click Create.

5. At the Select the audience for your task step, set up filters to select Tolokers for your pool.

   1. Clear My tasks may contain shocking or pornographic content if your project has none of those.

   2. To select Tolokers based on their language, location, age, gender, and other parameters, click the Add filter button.

      1. For example, add the Languages filter:

         

      2. It is best to launch voice recording tasks in the Toloka mobile app so that Tolokers can record audio directly in a task using the device's built-in voice recorder. Add the Client filter and select Toloka for mobile.

   3. Use the Speed/quality balance slider to change the number of Tolokers who can see your tasks. Move the slider to the right to exclude Tolokers with lower ratings from participating in your project.

      

6. At the Setup quality control step, set quality control rules for more accurate results:

   1. Click the Review task responses manually toggle, and specify the number of days for checking the task in the Review period in days field (for example, 21).

      What is manual review?

      The Review task responses manually option allows you to review completed assignments before accepting them and paying for them. If the Toloker didn't follow the instructions, you can reject the assignment.

   2. The Voice recording preset has the pre-configured quality control rules:

      • In most cases, you can keep the Fast responses rule as is. It filters out Tolokers who complete tasks too fast. The default settings mean that Tolokers are banned from the project for 1 day if they complete tasks in 4 out of 5 task suites in less than 15 seconds.

      • Delete the pre-configured Majority vote rule.

   3. For a trial pool, the settings you’ve just made are enough. You can get better results if you set the additional quality control rules.

      The additional quality control rules

      1. Add the Results of assignment review quality control rule and enter the following values:

         

         This means that if 35% or more of a Toloker's responses are rejected, the Toloker is banned and can't access your tasks for 15 days. The rule takes effect after 3 responses of the Toloker are reviewed.

      2. Add the Processing rejected and accepted assignments rule:

         

         This means that if you reject assignments during the review, they'll be sent for re-completion to another Toloker.

7. At the Set the task price and overlap step, set up how much a single task will cost for you.

   1. In Price per task suite, set the amount of money to pay per task suite done by one Toloker.

      A task suite is a page with a number of tasks. It can contain one or several tasks. If the tasks are simple, you can add 8–10 tasks per suite.

   2. In the Overlap field, define how many Tolokers must do each task.

      The value depends on how many recordings of the same phrase you want to collect. If one is enough, set 1.

   3. At the bottom of the Price section, you see Price per 1 task. This is the amount of money paid per task.

8. At the Add optional pool settings step, specify the Time per task suite, sec.

   This time should be enough to read the instructions and load the task (for example, 1,200 seconds).

9. At the Prepare and upload data step, upload your task data.

   1. Create the tasks for Tolokers:

      1. To download a template, click one of the buttons:

         • Template in XLSX
         • Template in TSV
         • Template in JSON

         For this type of project, the file with tasks must have one parameter. Its name equals INPUT:text, and the values are texts.

         INPUT:text
         Does white chocolate have cocoa?
         How many rings are there in the Olympic Games symbol?
         What is the capital of Australia?
         

      2. Open the downloaded file, and replace the sample values with your texts.

      3. Click Drop file here or select, and upload the file you’ve just made.

      4. Click Continue.

   2. Tasks are shown to Tolokers in suites. A suite is a single page with multiple tasks. Define how many tasks to include per suite:

      • General tasks: These are tasks for Tolokers to label.

      • Control tasks: These are tasks with predefined answers used to control the quality of responses. For this project, you don’t need control tasks.

      • Training tasks: These are tasks with predefined answers and explanations for Tolokers. Normally you use training tasks in separate training pools. You don’t have to include them.

      For example, you can add 5 general tasks per suite:

      

   3. Click Combine tasks into suites.

10. At the Double-check your project and try out tasks step, check how the task will look from the Toloker's point of view.

    Note

    This step will be enabled after you complete the previous steps. You can skip this step by clicking Do it later.

After all the steps, you'll see the Set up is finished and your pool is ready for labeling tip on the pool page.

Start labeling

1. Make sure you have topped up your account.

2. To send the tasks to Tolokers and begin the labeling process, click Start labeling.

   

3. In the pop-up panel, review the budget and click Launch.

See the results

Track the labeling progress on the pool page. You can start the review when the first results are received.

After the time period specified in step 4.1 of creating the pool, all responses are automatically accepted, regardless of their quality.

1. Go to the pool, and click Review assignments.

   

2. Choose an assignment.

3. Check the responses, and click Accept or Decline. For rejected responses, enter a comment to specify the reason.

   Note

   To learn about other ways of review, see the Reviewing Tolokers' responses section.

4. After checking all the assignments, click Download results.

   

   You will get the TSV file with the labeling results.

5. To download the recordings, click the arrow next to the Download results button. Choose Download attachments from the drop-down menu.

   

Troubleshooting

See also

• Crowdsourcing concepts: Instructions
• Crowdsourcing concepts: Quality control
• Crowdsourcing concepts: Pricing