Russian website launched in 2019 became the country's most popular online retailer
Over 200 million products for sale
Challenge
Required a scalable and effective product moderation system to comply with local cultural and legal specifics of CIS countries.
200 million items on the marketplace that require moderation
Products with legal restrictions pose a special risk to the buyer and the marketplace
Moderation involves difficult-to-define variables, such as cultural differences
Legal aspects change frequently and moderation must adapt
Solution
The method relies on a crowd of Tolokers to moderate items that the algorithms are not sure about.
Prioritize moderation of items that are displayed to users.
The ML model evaluates moderation items to determine if they are acceptable or should be banned from the website, and provides a confidence score for each class prediction.
Items with scores above the threshold are added to the database and passed on to the product service. Items with scores below the threshold are sent to Toloka for human evaluation.
All moderated items are evaluated independently by three annotators and approved when the majority of Tolokers agree on the classification.
After being sent to Toloka, the items are added to the learning set for the ML model to train on and improve the moderation threshold for future moderation.
Business results
The daily number of items moderated jumped from 200 to 100,000
The cost per moderated item dropped from $0.017 to $0.01
It now takes less than 15 seconds to verify one item
“Using instructional and scalable crowdsourcing allows us to estimate tricky things like insults and disgusting content. Getting this knowledge helps us train our ML model so that it also starts to understand. Even more importantly, we use these same annotations to improve our model, optimize it, and make the overall process cheaper. We now spend half as much on each item that needs to be moderated.”
– Elena Gruntova, Product Director, AliExpress
The case was presented at the Data-Driven AI meetup by Elena Gruntova, Product Director, AliExpress. The full video is available here.