Toloka documentation

Processing rejected and accepted assignments

You can enable the manual review option to reject or accept assignments that Tolokers completed. Assignments are rejected and accepted on a page-by-page basis.

Note

Once the review period in the Review period in days field runs out, tasks are accepted automatically.

When to use

This rule is helpful when you need to:

  • Resend rejected assignments for re-completion to other Tolokers.

    If you rejected an assignment, you may want it to be completed by another Toloker instead of the one whose response you rejected. To do this, you can increase the overlap for this assignment only. This is especially helpful if you have the overlap value set to 1.

  • Save money on re-completing assignments that you have already accepted.

    If you've reviewed and accepted a task, it may not make sense for other Tolokers to also complete it. To avoid this, you can reduce the overlap for accepted assignments only.

Rule settings

Warning

All fields in this rule are required. If you don't fill in at least one of them, you won't be able to save the rule.

Field

Overview

If

A condition for performing the action in the then field:

  • submitted assignments — The number of task suites that are completed and awaiting review.

  • approved assignments — The number of task suites accepted after the review.

  • rejected assignments — The number of task suites rejected after the review.

  • assignment becomes approved — The task suite was accepted after the review.

  • assignment becomes approved after rejection — The task suite was first rejected, but then accepted after the review.

    Note

    Note that you can't change the task status if the task pool was archived.

  • assignment becomes rejected — The task suite was rejected after the review.

To add multiple conditions, click .

then

Action to perform for the condition:

  • extend overlap by — Resend the task suite for completion to other Tolokers.

    If you want an assignment to be automatically reassigned even if your pool is already completed and closed, turn on the option Open pool if closed.

  • extend overlap by — Decrease the number of times a task suite can be completed by Tolokers. For example, use this action to cancel the recompletion of accepted assignments.

Rule example

The task is to send rejected and recompleted assignments for review. Or assign them to new Tolokers after rejection.

A Toloker's assignments rejected after the review are sent to another Toloker for recompletion.

A Toloker's assignments rejected after the review are not sent to another Toloker for recompletion.

For developers

Troubleshooting

What overlap should I set?

Overlap defines how many Tolokers complete the same pool task.

The best overlap is an overlap that provides satisfying quality of results. For most tasks that are not reviewed, overlap from “3” to “5” is enough. If the tasks are simple, overlap of “3” is likely to be enough. For tasks that are reviewed, set overlap to “1”.

Can I change overlap after the pool is started?

Yes. Open edit mode for the pool and set a new overlap value. You don't need to restart the pool. Updating the settings is usually fast, but if there are many tasks, it may take several minutes.

In this article: