What Is a Crowd Manager and Why Do They Have the Potential to Become Essential Employees?

Olga Megorskaya
by
Image

Subscribe to Toloka News

Subscribe to Toloka News

Experts often give talks at conferences about the professions of the future. The main message is that some of the skills that are in demand today will soon become obsolete. More and more tasks are being handled by AI. While this sounds cliché, the truth is undeniable. The same experts recommend hiring people who won't be replaced by robots over time: people who can design smart algorithms themselves. However, an AI developer is not the only profession of the future.

What is a crowd manager? The direct meaning «someone managing the crowd» accurately reflects the essence. This is an individual who meets business needs by assigning tasks to a large number of performers in a crowdsourcing system.

There are several well-known crowdsourcing platforms in the world, such as Amazon Mechanical Turk, Figure Eight, Lionbridge, and Toloka. Tens of thousands of performers complete a variety of small jobs on these platforms every day.

The challenge of a crowd manager is to organize the process so that the micro efforts of thousands of performers lead to fulfilling a high-level business task.

Which processes require a crowd manager?

Crowdsourcing is a powerful and flexible tool, and its scope is limited only by the imagination and abilities of the people who design crowdsourcing-based processes — the crowd managers. Let's look at a few examples.

How to get to know your client and make money off of it

Image
Data labeling for ML

Data labeling for ML

Crowdsourcing platforms can quickly and efficiently deliver a large amount of clean data for ML. For example, Toloka was originally designed as an infrastructure platform for providing Yandex ML projects with labeled data. Crowdsourcing is ideal for such tasks, because they require large amounts of labeled data with guaranteed quality.

The job of a crowd manager in these projects is to organize data labeling and quality control processes in a way that will involve as many labelers as possible. This is equally true for text markup, image segmentation, or audio transcribing.

Image
Selecting a design option

Selecting a design option

Let's say you have several designs to choose from for your product or ad. How do you decide which one to use? With crowdsourcing, you can digitize even subjective concepts like design appeal. A crowd manager prepares multiple tasks for comparing two objects, and it only takes a couple of hours to have several thousand workers vote on which one they like best.

You can also ask tolokers to make comments on their choice. Reading these comments can offer useful insights into how your product is perceived. Toloka tools help you target tasks like this at people from different categories to get a representative sample.

Image
Moderating content

Moderating content

Content moderation is a special area that requires fast decision-making and scalability, because if your website suddenly receives a barrage of comments, it's important to have enough moderators to keep up with the volume.

An experienced crowd manager doesn't need a staff of moderators to complete this task: it's optimal to carry this out with the help of the crowd. A crowd platform with a large number of tasks means that tolokers join your task when needed and switch to other tasks when there is nothing to do. If you reduce the moderation task to answering a number of specific questions and then automate the logic of decision-making based on these answers, anyone will be able to perform moderation tasks.

Image
Checking information in a certain location

Checking information in a certain location

An interesting use for crowd efforts is field tasks. We can ask people to perform small tasks when they are out walking around, like take a photo of the entrance to a local cafe or a shelf with products in a store.

This is not difficult for anyone to do, but if you collect enough of this data, you can solve complex business challenges, like remotely check if your store premises are clean in different locations, monitor grocery prices, check if entrances to organizations are accessible to the disabled, and much more.

This is a very nontrivial and interesting task: how can you ensure proper quality control for offline tasks that can be done anywhere in the world? Experienced crowd managers have the necessary skills to do exactly that.

What does it mean to manage the crowd?

Interestingly, despite a variety of potential applications, what a technical crowd manager does is clearly emerging as an independent profession. After all, regardless of the subject area, they use a common set of crowd management tools, including decomposition of a business task, organization of a production pipeline, quality control, and motivation of performers.

Let's look at each of them in more detail.

Decomposing and pipelining a business process is one of the key skills of a crowd manager. It's important to design a process so that the task is decomposed into a series of simple micro efforts that almost anyone can handle. This is how we get a system that thousands of tolokers are ready to join and contribute to at any time, and a crowd management specialist directs the energy of the performers towards completing the desired task.

Image
Decomposing a business process

Quality control is the second cornerstone of crowdsourcing, because no scalable system is helpful if it doesn't produce a quality result.

There are several standard approaches here:

  • Each task is assigned to multiple tolokers at the same time (overlap is used).
  • Tasks include control tasks that the system knows the correct response to in advance.
  • Non-automatic acceptance may be used for complex tasks. We pay a toloker who completes a task only after the result is verified and confirmed as high-quality. Needless to say, the verification process can also be set up as a pipeline using the crowd.

This way we get multiple signals from different Tolokers. After that the crowd manager's task is mathematical: to choose the model of aggregation of individual responses that will eventually give the best quality result.

The simplest model, referred to as majority vote, takes the verdict chosen by the majority and assumes it is correct. The majority is usually right, but not always. Sometimes you need to account for the individual expertise of more experienced Tolokers.

Image
Quality control

Motivation of Tolokers depends on having quality control set up properly: it is very important to reward people for doing a good job and use dynamic pricing to pay more for tasks done well.

What kinds of companies will need crowd managers?

The history of crowdsourcing and professional work with the crowd worldwide began in the 2000s when the Amazon Mechanical Turk crowdsourcing platform appeared and search engines started using assessors to rate quality and train search result ranking algorithms. Individual projects for using crowdsourcing appeared in other industries, too, but these were one-off and rather small-scale projects.

The revolution started in the last few years when ML technology began to permeate all business sectors on a massive scale. Two things became obvious: any ML is based on a well-labeled dataset that someone has to collect, and the time-tested methodology for working with the crowd on data labeling tasks can also be applied in other areas.

Image
Business sectors

Today, experts in crowdsourcing discover interesting ways to apply themselves in search engine companies, AI-related startups, banks, retail and other businesses. For example, Yandex engages tolokers to check business information on maps, moderate image content and find duplicates in Yandex.Images, assess translation quality in Yandex.Translate, mark up photos of streets for the Yandex self-driving car project, and perform hundreds of other tasks.

Since 2016, both Yandex services and our external customers can create tasks in Toloka. Currently, more than 40 thousand performers work daily on projects for Yandex and other companies, doing more than 15 million tasks a day.

The traditional role of the supervisor in such a system is disappearing: it's no longer a team lead for a few direct subordinates, it's a highly qualified engineer. Their purpose is to organize technology-based production processes involving tens, hundreds, and even thousands of people.

How to learn crowd management

There are three key skills that a professional crowd manager needs to have.

  • To transfer data from other systems to crowdsourcing platforms and between tasks within the platform itself, a manager must have basic programming skills. The most suitable languages are Python and C++. And knowledge of HTML, CSS, and JavaScript will help you design task interfaces.
  • It's also important to know statistics at the basic level (otherwise you won't be able to assess the quality of responses), choose the correct overlap models and other mechanisms.
  • Finally, the key skill of a crowd manager is to divide large tasks into small ones and formulate them in terms that are understandable to performers involved
Image
How to learn crowd management

To improve your skills, you need to regularly communicate with colleagues and share best practices. Yandex regularly holds meetings for everyone who wants to become a crowd manager.

In the beginning, it was about human skills that are considered valuable now, but that will lose relevance in the coming years due to the growing role of algorithms. Crowd management belongs to another group of competencies: these professionals won't disappear with the advent of AI but, on the contrary, will greatly affect its development.

Article written by:
Olga Megorskaya
Updated: 

Recent articles

Have a data labeling project?

Take advantage of Toloka technologies. Chat with our expert to learn how to get reliable training data for machine learning at any scale.
Fractal