Client

• Russian website launched in 2019 became the country's most popular online retailer
• Over 200 million products for sale

Challenge

• 200 million items on the marketplace that require moderation
• Products with legal restrictions pose a special risk to the buyer and the marketplace
• Moderation involves difficult-to-define variables, such as cultural differences
• Legal aspects change frequently and moderation must adapt

Solution

1. Prioritize moderation of items that are displayed to users.
2. The ML model evaluates moderation items to determine if they are acceptable or should be banned from the website, and provides a confidence score for each class prediction.
3. Items with scores above the threshold are added to the database and passed on to the product service. Items with scores below the threshold are sent to Toloka for human evaluation.
4. All moderated items are evaluated independently by three annotators and approved when the majority of Tolokers agree on the classification.
5. After being sent to Toloka, the items are added to the learning set for the ML model to train on and improve the moderation threshold for future moderation.

“Using instructional and scalable crowdsourcing allows us to estimate tricky things like insults and disgusting content. Getting this knowledge helps us train our ML model so that it also starts to understand. Even more importantly, we use these same annotations to improve our model, optimize it, and make the overall process cheaper. We now spend half as much on each item that needs to be moderated.”

– Elena Gruntova, Product Director, AliExpress

The case was presented at the Data-Driven AI meetup by Elena Gruntova, Product Director, AliExpress. The full video is available here.