Overview
To improve quality of answers, you must select the audience you need and train it.
Selecting the audience
Filters
-
Demographics (age, gender, education, languages, region, citizenship).
-
Device specs (device type, OS and browser version).
-
Top % of the best on the platform.
Training and onboarding
-
Correct answers + hints.
-
High scores continue on to the exam.
Exam
-
Scored by % correct answers.
-
Best scores grant access to paid tasks.
Multi-level quality checks
Quality control
-
Overlap (including dynamic overlap).
-
Validation by other annotators.
Anti-fraud system
-
Platform-wide ban for fraudulent Tolokers.
-
Behavior analysis system.
-
Multilayer technologies to detect and prevent all types of fraud.
Protect your project
Toloka has dedicated anti-fraud system for banning cheaters, but the quality control is shared responsibility of the requester and the platform. The requester is responsible for the quality control of his projects and protection of his data. Projects require individual approach in setting quality controls to ensure best quality of labelled data.
To protect your project from cheaters, you can use the quality control rules:
-
Ban for fast responses.
-
Limit skipped assignments.
-
Limit number of tasks per person.
Obtaining quality results after data labeling
Aggregating results
-
Using majority vote.
-
Using the Dawid-Skene method.
Evaluating metrics
-
Accuracy/completeness/F1/MCC, etc.
-
Consistency.
-
Confidence.
Reassigning tasks
-
If the submitted task is rejected.
-
If consistency is low.
-
If answers from banned Tolokers are thrown out.
Main project scenarios
Validate results
-
Control tasks — set the Toloker’s skill level based on answers in control tasks and exclude Tolokers who give the wrong answers.
-
Majority vote — have multiple Tolokers do the same task and look for consistency in answers.
-
Manually check results — evaluate Tolokers by the number of accepted and rejected tasks.
Diversify the audience
-
Earnings — limit the earnings per person in your pool in 24 hours.
-
Completed tasks — limit the number of tasks per person in your pool in 24 hours.
Prevent random clicking and bots
-
Fast responses — monitor the minimum time to complete a task suite.
-
Skipped assignments — exclude Tolokers who skip too many tasks in a row.
Redo certain tasks
-
Re-assign tasks completed by someone who was banned — if a Toloker gets banned, all their completed tasks can be automatically assigned to other people.
-
Rejected and accepted task processing — set the rules for assigning rejected tasks to other people.
Technologies for quality management
Transform the crowd into computing power with advanced technologies for quality management.
Multiple quality control methods
Toloka offers different approaches to achieve the best quality for each project.
-
Post-verification.
-
Task-based crowd training and testing.
-
Golden sets (honeypots) to monitor quality.
-
Advanced aggregation tools.
-
Platform-wide anti-fraud system.
Adaptive selection of Tolokers
Multi-stage selection of a distributed crowd.
-
Audience filters by language, age, gender, interests, location, real-time ranking, and more.
-
Training, exams, and retraining to find Tolokers for your exact task.
Smart matching mechanisms
Patent-pending matching system that honors the preferences of requesters and Tolokers for mutual benefit.
-
Invite Tolokers to a project who are most qualified to handle it.
-
Offer Tolokers personalized recommendations of interesting projects they will enjoy.
Autolabeling & verification (on demand)
Autolabeling and pretrained models with quality control built in.
-
Automated prelabeling. Results are verified by human Tolokers for high accuracy.
-
Human in the loop workflows.
What's next
- Set up quality control.
- Add tasks to the pool.
- Start the pool.
See also
Troubleshooting
Last updated:Â February 15, 2023