Screen annotators with Exams for higher-quality data

Make sure your experts are the right fit before your project starts, not after they've already touched production data. That's what exams are for.

Onboarding on the Toloka Platform is now split into two distinct steps: Training and Exam. Training is where experts learn your guidelines. Exam is where you confirm they actually understood them, before a single production task reaches their queue.

Why we built it

Projects have needed a self-service way to check expert quality before production work starts, and until now the options were limited. Simple exact-match checks work fine for classification labels, but they fall apart for open-answer tasks, bounding boxes, and other projects with complex outputs.

Bringing Training and Exam into one pipeline also means experts move from learning your guidelines to proving they understood them without extra friction in between.

How it works

Exams live inside your generation and quorum nodes, alongside your regular quality settings.

• Enable exams in the relevant section of the node

• Upload an exam dataset, or ask the agent to create one

• Select which entities you want to evaluate and how

• Set thresholds: how many tasks are required, and how many must be accepted to pass

Or skip the manual setup entirely and ask the agent to prepare the exam for you, dataset and configuration included.

Once an expert clears Training, they see a prompt to move straight into the Exam step, no detour back to a project list required.

Three ways to grade a submission

The part that makes exams useful for more than classification tasks is the grading logic. You're not locked into one method:

Exact ground-truth matching. Submissions are compared directly against reference values in your dataset. This is the closest thing to the old Toloka exam approach, and it's still the right call for classification labels and other structured outputs.

Instruction-based ground-truth comparison. For tasks like bounding boxes, where a reference answer exists but can't be checked with a direct comparison, the LLM QA agent uses a specific instruction to judge whether a submission matches the ground truth closely enough.

Production quality criteria. For open-answer tasks and other complex outputs, you can grade exam submissions against the same quality criteria your production pipeline already uses. Pick which criteria apply, and experts see a quality criteria panel so they know exactly what they're being measured against.

Because all three options exist side by side, one project can mix ground-truth checks for its structured fields with quality-criteria checks for its open-ended ones.

Training gets an upgrade too

Training now supports a practice step, where experts work through examples in the actual production UI rather than a static walkthrough. It's optional, but for anything with a learning curve, it's the difference between reading instructions and actually doing the task once before it counts.

Try it today

Open a generation or quorum node in any project, turn on exams, and set your first threshold. Or just ask the agent to build the exam for you and review what it comes back with.

Getting quality right at the door means fewer surprises once experts reach production, which is the whole point of putting an exam there in the first place.