Optimizing moderation of user-generated content

About the client

Our client, a moderation service, provides moderation solutions for various industries (marketplaces, geoservices, foodtech, and others) by relying mostly on automated moderation.

With an average of 5-15 million daily verdicts and automated responses for approximately 72-96% of cases, the moderation service plays a vital role in making online spaces safer and more secure for end users. Automated moderation includes:

• Performing real-time moderation for a limited set of verdicts

• Sharing anti-fraud verdicts for spam detection among users

• Building metrics for moderation speed, moderation quality, and percentage of unwanted content detected

• Training custom classifiers on an existing data stream to speed up moderation and cover more cases

The automated solution checks user-generated content for potentially harmful topics in text, images, and links. These include:

1. Text — profanity, insults, offensiveness, adult content, spam bot advertising, personal data, racism, extremism, toxicity, illegal activity

2. Images — adult content, violence, profanity, extremism

3. Links — malware, viruses, fraud, phishing, anti-spam bans, other security threats

The moderation system provides a set of low-level verdicts for these topics and rules for how to apply the verdicts by banning or allowing the content.

Challenge

Some content is too complex for automated methods to handle successfully. Up to 30% of tasks are labeled by ML models with a low confidence level and need more accurate labeling that humans can provide (average wait time of 5-15 minutes).

The client needed help from Toloka to:

• Improve output quality

• Minimize moderation costs

• Define and differentiate levels of violations

The client was looking to improve key business metrics for their moderation system: price per label, accuracy, recall, and labeling speed.

Solution

Tolokers manually verified ~5-30% of data sent to the service for moderation, providing round-the-clock human input and fast response times. They assigned verdicts for text, images, and links, so that the client could determine whether to allow or block the content and potentially ban the user.

Results

By using Toloka, the client reduced the average price per label 20 times compared to the alternative solution using in-house annotators.

With a focus on quality, speed, and efficiency, Toloka boosted the client’s metrics across the board, including the average accuracy and recall for projects related to text moderation.

Here’s how metrics improved:

• Average time for manual moderation: ~10-15 minutes for cases that could not be confidently assigned verdicts

• Average accuracy for projects related to text moderation: 91.5% (Toloka) vs. 74.8% (alternative solution)

• Average recall for projects related to text moderation: 94.6% (Toloka) vs. 83.1% (overall)