Demographics (age, gender, education, languages, region, citizenship).
Device specs (device type, OS and browser version).
Top % of the best on the platform.
Correct answers + hints.
High scores continue on to the exam.
Scored by % correct answers.
Best scores grant access to paid tasks.
Platform-wide ban for fraudulent Tolokers.
Behavior analysis system.
Multilayer technologies to detect and prevent all types of fraud.
Toloka has dedicated anti-fraud system for banning cheaters, but the quality control is shared responsibility of the requester and the platform. The requester is responsible for the quality control of his projects and protection of his data. Projects require individual approach in setting quality controls to ensure best quality of labelled data.
To protect your project from cheaters, you can use the quality control rules:
If the submitted task is rejected.
If consistency is low.
If answers from banned Tolokers are thrown out.
Control tasks — set the Toloker’s skill level based on answers in control tasks and exclude Tolokers who give the wrong answers.
Majority vote — have multiple Tolokers do the same task and look for consistency in answers.
Manually check results — evaluate Tolokers by the number of accepted and rejected tasks.
Completed tasks — limit the number of tasks per person in your pool in 24 hours.
Skipped assignments — exclude Tolokers who skip too many tasks in a row.
Re-assign tasks completed by someone who was banned — if a Toloker gets banned, all their completed tasks can be automatically assigned to other people.
Rejected and accepted task processing — set the rules for assigning rejected tasks to other people.
Transform the crowd into computing power with advanced technologies for quality management.
Toloka offers different approaches to achieve the best quality for each project.
Task-based crowd training and testing.
Golden sets (honeypots) to monitor quality.
Advanced aggregation tools.
Platform-wide anti-fraud system.
Multi-stage selection of a distributed crowd.
Audience filters by language, age, gender, interests, location, real-time ranking, and more.
Patent-pending matching system that honors the preferences of requesters and Tolokers for mutual benefit.
Invite Tolokers to a project who are most qualified to handle it.
Offer Tolokers personalized recommendations of interesting projects they will enjoy.
Autolabeling and pretrained models with quality control built in.
Automated prelabeling. Results are verified by human Tolokers for high accuracy.
Human in the loop workflows.
Last updated: February 15, 2023