Working with results
Aggregation
In the way it's calculated. In both aggregations, confidence means the same thing.
The method doesn't guarantee that original performer responses will be used for aggregation. The algorithm takes into account performers' quality parameters and response patterns. Consequently, it can return a result that's different from the performers' responses to this task.
The pool page contains the List of Operations button.
You cannot aggregate by project fields that have no valid values. Specify the possible values for all the fields of all types.
You need to use smart mixing.
Processing the results file
When you upload a file with rows, double quotes indicate an area where you can use special characters (tabs or line breaks). Toloka merges everything in between the quotes into one row to make up one task. To use double quotes inside such an area, you need to escape them with another quote. Read more here.
null
, otherwise, it's undefined
.Assignment review
- Assign them a hidden skill, or use an existing skill linked to the pool.
- Go to Messages and click Write → Group → Add filter → Choose your skill → <skill name>.
- If you created a new skill, specify the value you assigned to the chosen group, (for example, 1). If you use an existing skill, specify the minimum value.
You can only accept or reject an assignment.
You can deny access to the pool if the performer's responses are too fast, if they don't match the majority vote, or if the performer makes too many mistakes in control tasks. Tasks completed by such performers can be given to other performers.
No, you can't fix anything in the task itself. However, you can do this manually in the results file.
Simply give the performer a separate reward without changing the task status. You can't change the task status in the pool in this case.
Accept the task and update the instructions. Otherwise, you violate the Requester Agreement that requires to clearly state the task requirements and the results expected from the user.
No. For example, there are 10 tasks in a suite that costs $0.10, and the performer did 2 of them incorrectly.
You can't accept the correct answers and pay for this part ($0.08). Response pages are accepted or rejected in their entirety.
Perhaps the performers found your tasks too difficult. Try to simplify them.
Shorten instructions and rewrite using simpler language. Add pictures and examples.
Make the interface more user-friendly and don't make the performer complete unnecessary actions. Use keyboard shortcuts.
Starting from December 15, 2021, tasks in pools are automatically available in the web version of Toloka and the mobile app. If your task isn't available on mobile devices, it means that you might have forgotten to add the Client filter.
Reply to messages from performers regularly. Try to provide feedback as fast as possible. Correct errors promptly and use mailing lists to notify performers of changes.
Statistics
The percentage is calculated based on the total number of pages, including the overlap. If the pool uses smart mixing with control tasks, the maximum progress bar value and the currently completed percentage are calculated approximately.
“Quality: training tasks” includes only the training pools linked to the main pools, but the control and training tasks uploaded to the main pools are counted in “Quality: control tasks”.
When you download the results file, select the Start time and Submit time checkboxes. The downloaded file will include the exact UTC date and time when the performer accepted the assignment and submitted it.