Every requester wants to optimize expenses. This section describes the main ways to save money without reducing the labeling quality.
Use decomposition to simplify the tasks:
Dynamic overlap lets you change overlap depending on:
With this overlap setting, you can save on the number of Tolokers who complete your tasks, which means the total cost of pool labeling will be lower.
Manual review allows you to check responses so you don't have to pay for poorly completed tasks.
Use manual review if:
The task requires that users send free-format responses or data files.
You are sure that you have enough time to review the Tolokers' responses. Otherwise, when the period ends, responses that haven't been reviewed are accepted and paid for automatically.
You can set up a selective majority vote check for the general tasks. This helps you save money and speed up pool completion. You can set the period during which tasks with increased overlap will be displayed to the Toloker, depending on the number of completed tasks.
Dynamic pricing lets you increase the price per task suite for Tolokers with a higher skill level. This way you can encourage Tolokers to do a better job.
In some cases, this setting makes labeling more expensive, like for tasks that don't require Tolokers to have high skills. In this case, there is no need to use dynamic pricing.
Use the Smart mixing method and the Allow partial task suites option. This way you can save money on paying for incomplete task suites.
In Toloka, requesters pay for exams, including those that Tolokers didn't pass. This is a fee that a requester doesn't get any benefit from, because a Toloker failed the exam and won't be able to complete any of the general tasks.
In this case, you can set up automatic acceptance of tasks and automatic rejection of tasks at the end of the review period. To do this, you'll need to:
Select Exam as the pool type.
Turn on the Review task responses manually option.
Turn on the Reject tasks at the end of the review period option and specify the reason for rejection. It will be shown to Tolokers whose responses are rejected.
Set the manual review period to one day.
Add the Control tasks quality control block and set the If conditions:
correct responses (%) ≥ X;number of responses = Y.Select the accept all assignments from this Toloker in the pool value in the then field.
Add a description of the new rule to the instructions.
Compress images so that a task opens and displays correctly. Tolokers will spend less time and traffic uploading images.
Create keyboard shortcuts. This will make it easier for Tolokers to complete your tasks. Remember: the less time it takes to complete a task, the less you pay for it.
You can merge tasks that have identical input data. This allows you to save money if the same task was uploaded to different pools.
Using the Toloka API, you can set the frequency of assigning control tasks. For example, make every fifth task a control task in the first task suite and every 10th task in the following task suites. This setting lets you maintain the quality of the pool and speed up the completion of tasks.
To do this, use the golden_task_distribution_function key.
In addition to dynamic overlap, use baseline_solutions to speed up the overlap. In baseline_solutions, preliminary responses are used, that is, data that simulates Toloker responses. For example, you need to determine what an image shows: a cat or a dog. Suppose your neural network determined that the image may show a dog with a probability of 80% and a cat with a probability of 40%. You can use the neural network's responses. Depending on the responses of Tolokers, the overlap will increase or remain unchanged.
Last updated:Â August 3, 2023