Working with results

What is the difference between the confidence in the aggregated response in the Dawid-Skene aggregation model and the confidence in aggregation by skill?

In the way it's calculated. In both aggregations, confidence means the same thing.

How does the Dawid-Skene aggregation model work?

The Dawid-Skene aggregation model analyzes the Toloker responses and creates a confusion matrix for each Toloker. This lets us evaluate the statistical significance of the Toloker in the context of each assignment. Learn more about the model.

Why does the Dawid-Skene aggregation model return a result that the Tolokers didn't select?

The method doesn't guarantee that original Toloker responses will be used for aggregation. The algorithm takes into account Tolokers' quality parameters and response patterns. Consequently, it can return a result that's different from the Tolokers' responses to this task.

Where do I see the aggregation progress?

The pool page contains the List of Operations button.

Why might aggregation by Toloker skill be unavailable?

You cannot aggregate by project fields that have no valid values. Specify the possible values for all the fields of all types.

You can't aggregate by skill. When running via the API, I get the error code ONLY_FOR_POOL_WITH_MIXER. Why?

You need to use smart mixing.

Can I get notifications when results aggregation finishes?

Yes. To set up notifications in your account, go to Profile → Notifications → Pool or aggregation completed. Learn more about setting up notifications.

Other questions

Why do I get blank spaces inside my TSV file?

When you upload a file with rows, double quotes indicate an area where you can use special characters (tabs or line breaks). Toloka merges everything in between the quotes into one row to make up one task. To use double quotes inside such an area, you need to escape them with another quote. Read more here.

What is the result of processing an empty text field?

If a Toloker typed some text and then deleted it, the result is null, otherwise, it's undefined.

Other questions

How do I send an assignment back to the Toloker for revision?

You can only accept or reject an assignment.

Can I disable tasks for Tolokers who do a poor job on tasks?

You can deny access to the pool if the Toloker's responses are too fast, if they don't match the majority vote, or if the Toloker makes too many mistakes in control tasks. Tasks completed by such Tolokers can be given to other Tolokers.

Can I fix something in a completed task myself?

No, you can't fix anything in the task itself. However, you can do this manually in the results file.

What should I do if I want to accept a completed task but the pool is already archived or the Toloker wrote to me after the allowed time?

Simply give the Toloker a separate reward without changing the task status. You can't change the task status in the pool in this case.

What should I do if I rejected a task for a reason that isn't specified in the instructions?

Accept the task and update the instructions. Otherwise, you violate the Requester Agreement that requires to clearly state the task requirements and the results expected from the Toloker.

Can I reject part of the responses on the page and accept part of them?

No. For example, there are 10 tasks in a suite that costs $0.10, and the Toloker did 2 of them incorrectly.

You can't accept the correct answers and pay for this part ($0.08). Response pages are accepted or rejected in their entirety.

How can I notify the Toloker of changes in the instructions?

Add the notification to the project description (for example: “Attention! The instructions changed”) and send a message to all the people who completed your tasks. To do this:

• Assign them a hidden skill, or use an existing skill linked to the pool.
• Go to Messages and click Write → Group → Add filter → Choose your skill → <skill name>.
• If you created a new skill, specify the value you assigned to the chosen group, (for example, 1). If you use an existing skill, specify the minimum value.

How can I increase the project's rating?

The project rating is the average rating across all categories. Pay particular attention to the categories for which you got the least points.

Low rating for “Will you take similar tasks in the future”

Perhaps the Tolokers found your tasks too difficult. Try to simplify them.

Low rating for “Clarity of instructions”

Shorten instructions and rewrite using simpler language. Add pictures and examples.

Low rating for “Task interface usability”

Make the interface more user-friendly and don't make the Toloker complete unnecessary actions. Use keyboard shortcuts.

Starting from December 15, 2021, tasks in pools are automatically available in the web version of Toloka and the mobile app. If your task isn't available on mobile devices, it means that you might have forgotten to add the Client filter.

Low rating for “Communication with the requester”

Reply to messages from Tolokers regularly. Try to provide feedback as fast as possible. Correct errors promptly and use mailing lists to notify Tolokers of changes.

If you don't understand what the problem is, run a mini-survey and ask the Tolokers who completed your tasks what they like and what they don't like.

Other questions

What is the formula for calculating the percentage of pool completion?

The percentage is calculated based on the total number of pages, including the overlap. If the pool uses smart mixing with control tasks, the maximum progress bar value and the currently completed percentage are calculated approximately.

What happens to the tasks that have the expired or skipped status?

They return to the pool until the full overlap is reached.

How are the statistics for “Quality: training tasks” and “Quality: control tasks” calculated? Do they include the training tasks uploaded to the main pool?

“Quality: training tasks” includes only the training pools linked to the main pools, but the control and training tasks uploaded to the main pools are counted in “Quality: control tasks”.

Can I get the exact start and end time from the Toloker with the results of the task?

When you download the results file, select the Start time and Submit time checkboxes. The downloaded file will include the exact UTC date and time when the Toloker accepted the assignment and submitted it.

Other questions