Toloka has dedicated anti-fraud system for banning cheaters, but the quality control is shared responsibility of the requester and the platform. The requester is responsible for the quality control of his projects and protection of his data. Projects require individual approach in setting quality controls to ensure best quality of labelled data.
Use quality control rules to get better results for tasks and restrict access to fraudulent users. Rules are configured for both projects as a whole (the Quality control tab) and individual pools.
If your project has a lot of similar tasks, you'll need to protect it from fake Tolokers (robots) and cheaters.
Use the Fast responses rule to weed out Tolokers who perform tasks too quickly and carelessly. You've tried doing tasks and you know that each task takes at least 15 seconds. You should suspend Tolokers who submit three out of ten assignments in a row faster than the control time.
Another “optimization” technique that cheaters use is skipping certain assignments. They pick out the ones that seem easier to them, while other Tolokers reliably complete all the assignments they receive. However, both get paid in the same way.
Use the Skipped assignments rule to restrict access to Tolokers who skip several pages of tasks in a row.
Limits are also useful when you need as many different Tolokers as possible. For example, to collect data for statistics, run surveys, or pick the best version of a logo. Another case is personal descriptions for a dating service. If texts are all written by the same person, they are likely to turn out monotonous and boring.
You can limit the number of tasks in the pool per Toloker or set a limit on daily earnings for your project. Use the Submitted responses and Earnings rules.
Results from conscientious Tolokers should be checked, too. In Toloka, this can be done automatically.
Tolokers complete tasks by selecting one of multiple possible answers. To check the quality in this type of project, use the Control tasks rule. This involves adding questions where you define the correct task response.
Mix the control questions in with the general tasks. They should make up at least 1% of the total tasks. Assign Tolokers a skill level depending on how many answers they get right.
The skill filters will remove anyone who makes a critical number of errors. Pay highly skilled Tolokers more. That will encourage everyone to try harder.
Sometimes requesters aren't able to regularly add control tasks and keep them updated.
In this case, the Majority vote rule is helpful. The platform assigns the same task to multiple Tolokers. For example, five people get the exact same task. If three or more choose the same answer, we assume it is correct.
Use the percentage of matching task responses to assign skill levels, control access to tasks, and reward Tolokers.
For tasks that aren't multiple-choice, the customer or other Tolokers check the answers. They might include recording audio, taking pictures, or writing something. Poor-quality task suite responses are rejected.
Use the Review results rule to control Tolokers' access to tasks. The platform calculates the percentage of task suite responses accepted and rejected for each Toloker. Those who often made mistakes will stop getting tasks.
Quality control rules let you minimize the consequences of errors. Toloka lets you send individual tasks back to Tolokers to be redone or have them redo all their work if they make too many mistakes.
For example, you can use Toloka to improve speech recognition algorithms. Tolokers listen to a series of 5–10-second recordings and make a transcription of each of them in order to convert speech to text.
In some cases, they get it wrong. Use the Processing rejected and accepted assignments rule to automatically have Tolokers review each other's work.
When Tolokers make too many mistakes, the platform bans them from taking on more tasks. But the task suite responses that they have already provided remain in the database and may get into the final dataset. Send these assignments for recompletion.
The Recompletion of assignments from banned users rule is helpful here. While the cost will go up, there will be fewer errors.
The best practice is to combine quality control rules. For example, if tasks involve selecting a task response from several options, you should use:
Content generation projects like writing texts or taking photos need a different combination:
Each project requires its own approach. Methods that are effective in one case may be pointless in another. But there are some general recommendations:
Analyze the task and select the appropriate quality control rules.
Write clear instructions. If you create control tasks, make sure they don't have any outdated, ambiguous, or incorrect responses.
Test your project in the testing environment, the sandbox. Running and completing tasks in it will help you understand the rules and estimate how much time it will take to complete a single task.
Launch the first real project with a small number of tasks. Analyze the results and adjust the settings if necessary.
Monitor the labeling results and listen to feedback from Tolokers.
Contact support If something isn't working.
Last updated: March 2, 2023