Creating a file with tasks

• Add another quotation mark to each " type of quotation mark. Don't escape other types of quotation marks (« » and “ ”).

• Enclose the field in quotation marks " ".

• Add another quotation mark " or a backslash and a quotation mark \" if there is no backslash before the quotation mark.

• Enclose the field in quotation marks " ".

• Escape it with an additional slash \.

• Enclose the field in quotation marks " ".

• Add another quotation mark to each " quotation mark. Don't escape other types of quotation marks (« » and “ ”).

• Add a backslash \ before each comma inside the object if it isn't there already. Don't escape commas that separate objects inside the array.

• Enclose the field in quotation marks " ".

• Add another quotation mark " or a backslash and a quotation mark \" if there is no backslash before the quotation mark. Don't escape other types of quotation marks (« » and “ ”).

• Enclose the field in quotation marks " ".

• Escape it with two backslashes \\.

• Enclose the field in quotation marks " ".

The number of tasks depends on how difficult and time-consuming the tasks are. Keep the size reasonably small. Large task suites are unpopular, partly because they are inconvenient for Tolokers (for example, if the internet connection is unstable).

Errors in column headers

If the column headings are incorrect, the whole file is rejected. Otherwise, Toloka specifies the number of tasks with processing errors.

Processing errors tables

"parsing_error_of": "https://tlk.s3.yandex.net/wsdm2020/photos/2d5f63a3184919ce7e3e7068cf93da4b.jpg\t\t",
"exception_msg": "the nameMapping array and the sourceList should be the same size (nameMapping length = 1, sourceList size = 3)"

"exception_msg": "the nameMapping array and the sourceList should be the same size (nameMapping length = 4, sourceList size = 6)"

"code": "VALUE_REQUIRED", "message": "Value must be present and not equal to null"

"code": "INVALID_URL_SYNTAX", "message": "Value must be in valid url format"

"exception_msg": "unexpected end of file while reading quoted column beginning on line 2 and ending on line 4"

You can specify the number of tasks on the page when you upload your tasks to the pool. For more information about distributing tasks across pages, see this article.

Use the button Upload review results to upload your file. You can see the format here.

Assignments are reviewed in the tasks file.

Check the hint field. For the general tasks, this field must be empty.

In the file with the general tasks, the columns with the INPUT headers must be filled out. You can see those headers if you download a sample file from the pool.

If you are creating control tasks, fill out the GOLDEN columns with the correct responses.

If you are creating a training task, you also need to fill in the HINT:text column. For the general tasks you don't need any columns other than INPUT, so feel free to delete them.

You have to use the TSV, XLSX or JSON file format and UTF-8 encoding.

For more information about creating the file, see the Guide. If there are errors during the upload, look up the error description on this page.

The error might occur if the expected input type is URL, but a string is received.

There may be two reasons:

• The input field has the "link" type.

• The pool was created for an outdated project version. It means that the pool was created before you changed the input field type.

It depends on the task. Technically, you can use as many tasks you want.

But Tolokers don't like to take lengthy tasks. They'd rather do 10 tasks that take one minute each than one task that takes 10 minutes.

In addition, if you use a large number of tasks on the page, there might be issues with uploading the files to be labeled. This problem might occur with images.

The third thing to consider is quality control and manual review. If you allow recompletion of assignments by banned Tolokers, you should split the task into smaller parts so that fewer assignments are recompleted. You are more likely to meet your budget this way.

Smart mixing settings are specified for the file rather than for the pool.

The settings specified during the first file upload are applied to all the files that are uploaded to this pool later on.

Try completing the tasks yourself. Ask your colleagues and friends to complete them. Find out average completion time and add 50% to it.

A task means a separate task. A task suite means a page with tasks. The Toloker gets paid for a task suite.

The same task may appear on different pages if:

• Dynamic overlap is used (incremental relabeling, IRL). As an example, let's say there were 5 tasks on a page. For 4 of them, responses coincided and the common response was counted as correct. The fifth task was mixed into another set because it didn't get into the final response and it needs to be “reassessed”.

• Different tasks have different overlap. Tasks with higher overlap will be additionally shown in sets with the other remaining tasks in the pool.

• If a quality control rule changes a task's overlap, it will appear in a different set.

You must use a separate row for each task in your tasks file. For more information, see here.

When you create a pool, the pool will have settings for the number of tasks per suite.

You can't use the interface to upload the tasks with multiple correct responses to the pool. You can only use the API for that.

If you have the Keep task order option enabled in the pool settings, labeling will start after the tasks you uploaded previously are taken by Tolokers. If this option is disabled, we can't guarantee that the tasks will be assigned in that order.

All the values are written to the same column. Make sure to escape quotes. For more information about escaping quotes in JSON format, see the Guide.

The array of strings in the input data must be comma-separated. For example: INPUT:typestext1, text2, text3, text4

The order of the fields in the file does not matter. Use your preferred order of fields.

No. If you try to upload a file with missing headers to the pool, the system issues an upload error. All the INPUT fields required in the specification must be present in the tasks file. There must be no extra fields or columns.

If you don't want to show some data to Tolokers, but you still need this data in the file, create the optional hidden input fields for such data in the project.

Escape commas with a backslash (\).

The text from the hintfield will be shown to the Toloker in a red box at the top of the page if they give a response to the control task that differs from the correct one.

Example

If you need to display the text from the hint field in several lines, add hyphens to the file and enclose the text in quotation marks.

"Add your text here" is a hint for you. It means that you can replace the text in the field with your task data. The file structure and how to fill it out is described here.

If you have one word enclosed in quotes, format the uploaded assignment like this: "How many letters are there in the word ""Liechtenstein""". If you are escaping quotes inside your text, then the entire text must be enclosed in quotes. For more information, see the Guide.

Text in the GOLDEN field must match the control text exactly.

Usually, if you copy site links from the browser, the copied links have the same format. But this is not the case when the link is trimmed or typed manually.

Check the links that you use. There are several ways to unify links:

• Add requirements for the link format in your instructions and hints in your training pool.

• Use RegExp in your JS to trim the received links and write the result to the new output field, and then match the received value against the control value.