Receiving responses
You can download Tolokers' responses in a file as tasks are completed.
If tasks were distributed with an overlap of more than 2, run aggregation. Toloka will process all Tolokers' responses for the task and form the resulting response.
Getting the file with responses
In the interface
To get a file with Tolokers' responses, click the Download results button on the pool page.
Alert
The responses submitted by the banned Toloker before the ban will be taken into account and paid for. To exclude their responses from the results and aggregations, select the option Exclude assignments from banned users. It will delete the responses from users who were banned at the moment of downloading the results, not when the pool was labeled.
The fields in the file with responses:
-
INPUT:<name of the input data field>— Input data for tasks. -
OUTPUT:<name of the output data field>— The Tolokers' responses. For training tasks, the field includes first attempted answers. -
GOLDEN:<name of the output data field>— Responses for control tasks. -
HINT:text— Hints for training tasks. -
Information about the task completion (the same for all tasks on the page):
-
ASSIGNMENT:link— Link for viewing the task suite. -
ASSIGNMENT:assignment_id— ID of the assigned task suite. -
ASSIGNMENT:worker_id— ID of the Toloker who completed the task. -
ASSIGNMENT:started— Date and time the task suite was assigned. -
ASSIGNMENT:status— Task status:SUBMITTED— completed,APPROVED— accepted,REJECTED— declined.
-
A sample file:
How to work via Toloka API
To get the list of the responses using Toloka API, send a GET request with either pool_id, task_suite_id, or task_id specified:
curl -X GET 'https://toloka.dev/api/v1/assignments?pool_id=1085757' \
-H 'Authorization: OAuth AQC2AGAJgyNSA8CtpdO9MWy_QEB6s6kDjHUoElE'
Refer to the Get list of responses section of the Toloka API documentation for more details about the request, its parameters, and possible responses. You will find examples of the requests in Toloka-Kit and other code samples there.
Getting files
If the Tolokers were asked to upload files, the file with responses will contain IDs of the files received from Tolokers. To download files on your computer, click the button on the pool page.
You can download all files in the pool in one archive up to 4 GB in size. If the archive is too large (more than 4 GB), use the following tips:
-
Select a single status or pick a smaller time range.
For example, if you need to review assignments, only download the submitted assignments rather than the accepted or rejected ones.
If the file still exceeds the permitted size, pick a smaller time range and download the archives in several batches. The time range can be reduced to 1 day.
-
On the pool page, click Review assignments (or View assignments if they have already been accepted) to go to the page with the Tolokers' responses. Each string contains the Toloker's response to one task suite.
Select the responses you need and click . Download the archives in several batches. You can select and download no more than 100 responses at a time.
-
Use the Toloka API to download files with responses.
See also
For developers
- Toloka API: Getting responses
- Toloka API: Files in responses
- Toloka-Kit recipe: Get list of responses
- Toloka-Kit recipe: Get list of files in responses
- Toloka-Kit recipe: Download attachments
Troubleshooting
No. After sending a task, the Toloker can't make any changes to it. You can add tasks that were completed incorrectly to a new pool.
No, you can't fix anything in the task itself. However, you can do this manually in the results file.
You can only accept or reject an assignment.
You can deny access to the pool if the Toloker's responses are too fast, if they don't match the majority vote, or if the Toloker makes too many mistakes in control tasks. Tasks completed by such Tolokers can be given to other Tolokers.
Simply give the Toloker a separate reward without changing the task status. You can't change the task status in the pool in this case.
Accept the task and update the instructions. Otherwise, you violate the Requester Agreement that requires to clearly state the task requirements and the results expected from the Toloker.
No. For example, there are 10 tasks in a suite that costs $0.10, and the Toloker did 2 of them incorrectly.
You can't accept the correct answers and pay for this part ($0.08). Response pages are accepted or rejected in their entirety.
Add the notification to the project description (for example: “Attention! The instructions changed”) and send a message to all the people who completed your tasks. To do this:
-
Assign them a hidden skill, or use an existing skill linked to the pool.
-
Go to Messages and click Write → Group → Add filter → Choose your skill → <skill name>.
-
If you created a new skill, specify the value you assigned to the chosen group, (for example, 1). If you use an existing skill, specify the minimum value.
The project rating is the average rating across all categories. Pay particular attention to the categories for which you got the least points.
Perhaps the Tolokers found your tasks too difficult. Try to simplify them.
Shorten instructions and rewrite using simpler language. Add pictures and examples.
Make the interface more user-friendly and don't make the Toloker complete unnecessary actions. Use keyboard shortcuts.
Starting from December 15, 2021, tasks in pools are automatically available in the web version of Toloka and the mobile app. If your task isn't available on mobile devices, it means that you might have forgotten to add the Client filter.
Reply to messages from Tolokers regularly. Try to provide feedback as fast as possible. Correct errors promptly and use mailing lists to notify Tolokers of changes.
If you don't understand what the problem is, run a mini-survey and ask the Tolokers who completed your tasks what they like and what they don't like.
When you upload a file with rows, double quotes indicate an area where you can use special characters (tabs or line breaks). Toloka merges everything in between the quotes into one row to make up one task. To use double quotes inside such an area, you need to escape them with another quote. Read more here.
If a Toloker typed some text and then deleted it, the result is null, otherwise, it's undefined.
The results show the responses of all users, including those who are banned. To exclude their responses from the results, select the option Exclude assignments by banned users. It will delete the responses from users who were banned at the moment the results were downloaded, not when the pool was labeled.
When you download the results file, select the Start time and Submit time checkboxes. The downloaded file will include the exact UTC date and time when the Toloker accepted the assignment and submitted it.