Manual review

Manual review is a pool setting that allows you to check responses and not pay for poorly completed tasks. You can review responses manually or with the help of Tolokers by posting a separate review task.

Learn more about manual review methods.

How does it work?

You set up manual review for your pool in the project.

As the Tolokers complete the tasks, their responses come for review. You can accept or reject a Toloker's response. If you reject responses, the Toloker can:

  • Challenge the review result by writing a message to the requester — submit an appeal.

  • Write to the platform support service.

  • Put a low rating to your project.

How to get fewer appeals and maintain a high rating for your projects
How to get the review statistics
Pool status and the review results

Rejecting responses

  • If you reject responses, assignments aren't sent automatically to other Tolokers.

  • You can't reject or accept one response on the page — only the entire page with all responses on it.

  • If the Toloker is banned by the quality control rules, their responses still need to be checked.

Note

Rejecting the Toloker's responses in one task doesn't result in rejecting responses of this Toloker.

Automatic rejection and acceptance of responses

If your pool type is set to Exam or Retry, then at the end of the review period, all responses that haven't been reviewed can be accepted or rejected automatically.

Automatic rejection

When setting or editing the pool, enable the Reject tasks at the end of the review period option and specify the reason for rejection. It will be shown to Tolokers whose responses are rejected.

Example

Automatic acceptance

Leave the Reject tasks at the end of the review period option turned off, and the tasks will be automatically accepted.

Example

How do I turn on manual review?

Take the following steps before starting the pool:

  1. Choose the Review task responses manually parameter on the pool page in edit mode.

  2. Specify the Review period in days — the number of days for the review (from 1 to 21 days from the task completion date). The Toloker will see the deadline in the task information on the main Toloka page.

    Note

    Set the review period up to 14 days. Tolokers are reluctant to accept assignments with a longer review period.

  3. Click the Save button at the bottom of the pool parameters page.

As the Tolokers start completing the tasks, you will start receiving responses.

Warning
  • Set the deadline so that you have enough time to review the responses. After the deadline expires, the responses that haven't been reviewed are accepted and paid automatically, except for the cases described above.

  • Explain the acceptance criteria in the instructions for Tolokers. This will improve the quality of the results, as well as reduce the number of appeals and questions about rejection reasons from the Tolokers.

More settings

Send tasks with rejected responses for re-check
Filter out Tolokers if their responses are often rejected

What's next

Contact support

Last updated: July 7, 2023

Introduction
Getting started
Important tips
Useful recommendations
Working with Toloka
Projects
Pools
Tasks
Results
Project analysis
Toloka settings
Task interface
Template Builder
HTML/CSS/JS editor
Help and support
FAQTroubleshootingSupportGlossary