This method works well if you need to review a lot of responses, filter the results first, or process them programmatically. Use it to filter out meaningless or senseless comments, calculate the majority vote, or ignore responses submitted by cheating performers.
- Download the TSV file with responses from the pool page.
- For the ones you rejected, add a comment explaining the reason in ACCEPT:comment. accepted responses and "-" for rejected responses.
- For the ones you rejected, add a comment explaining the reason in ACCEPT:comment.
If each task page (suite) has multiple tasks, they'll have the same ASSIGNMENT:assignment_id. This is the unique ID of the page with responses, so you first need to decide whether to accept the entire task suite. Add the plus or minus sign for each task on the page, count the total, and then leave only one line per assignment_id in the file. If there are more correct responses in the task suite than incorrect ones, accept it. Otherwise, reject it.
- Upload the prepared TSV file with the verdicts to Toloka and you'll have the responses reviewed.
Make sure to remove extra lines. If you upload a file with duplicate assignment_ids, the system will randomly select one of these lines and apply its verdict to the entire task suite, either accepting or rejecting it. In this case, after processing the file, some tasks may get a status different from the one you expected.