Performance indicators: ensure the quality of data labeling projects

Toloka Team
by Toloka Team
Image

Subscribe to Toloka News

Subscribe to Toloka News

Sometimes you set up a data labeling project but it doesn't go as planned — we feel your pain. You want to make sure you've done everything right before you spend your budget. Our new project performance indicators can help you quickly see where your project needs tuning to get the optimal speed and quality of results.

What are project efficiency indicators?

Efficiency indicators are metrics tracked during project execution. They show how different project settings affect labeling quality and speed. This way, you can evaluate project effectiveness and determine what changes are required to improve labeling results for your project.

How does it work?

Efficiency indicators are calculated for each pool in the project and then displayed on the Efficiency tab of the pool page. By viewing and analyzing all the metrics in one place, you can easily identify risks and areas for improvement.

Image

When Toloka detects a problem, it issues a risk warning. A caution icon appears next to the project in the project list and on the pages of individual projects and pools. By clicking this icon, you can learn how the issue might affect your results and how to fix it.

Image

What kind of data is tracked?

The Efficiency indicators for your projects and pools show how your settings affect two main labeling parameters:

  • Quality. Affected by availability of control tasks.
  • Speed. Affected by available Tolokers, bans, and submission rate.

Some indicators are based on the reviews and ratings submitted by Tolokers:

  • Quality of instructions.
  • Task interface quality.
  • Communication issues with the requester (will appear later).

You can find out if Tolokers are struggling with any part of your project and fix your instructions or task interface as needed.

Can I view the data used to calculate the indicators?

The interface displays information about how each indicator is calculated and what needs to be done to resolve the issue. To learn more about each indicator, check out the Toloka Help.

Give it a try

Want to try it out? Leave a request here.

Request new feature

If you have any suggestions for additional metrics that you'd like to see for your projects, let us know — we're always happy to hear your ideas.

Article written by:
Toloka Team
Toloka Team
Updated: 

Recent articles

Have a data labeling project?

Take advantage of Toloka technologies. Chat with our expert to learn how to get reliable training data for machine learning at any scale.
Fractal