Moderating public chats in social media: faster results, lower costs​​

About the client

Our client is a content and news sharing platform with more than 70M monthly active users. One of the popular features is public group chats, where users can share text messages, images, and videos. All user-generated content needs to pass moderation to make sure it complies with the platform’s guidelines.

Challenge

The client was already using an internal workforce to handle content moderation. As the platform grew in popularity, this workforce couldn’t keep up — they needed a scalable solution to moderate non-compliant public channels by monitoring messages and threads​. The goal was to ensure a safe environment and protect users from fraud.

Solution

Toloka developed adaptive ML models for automated moderation in 7 classes and 3 languages — English, Russian, and Ukrainian.

The solution detects these classes in text, images, and videos:

• Offensive speech

• Clickbait

• Money transfer

• Spam

• Selling activity

• Personal data

• Illegal content

We use a human-in-the-loop pipeline to ensure quality. Toloka’s crowd of qualified annotators evaluate the accuracy and precision of model output and provide feedback for additional model training to continually improve the model output.

If a user reports inappropriate content, it is sent directly to Toloka for manual moderation. Annotators label the content with one of the categories above, or mark it as compliant. Depending on the verdict, the content is removed from the platform or approved.

Business impact

Toloka’s solution detects 30% more non-compliant content than the previous in-house solution, processing 10-20K messages per day. The automated solution is much faster, taking about 5 minutes to verify one item. Moderation costs dropped by 40% for the client. At the same time, user retention increased by 20% due to better user experience with less fraud.

Toloka Moderation offers out-of-the-box solutions and enterprise customization for efficient content moderation to protect your users.