Improving taxi customer satisfaction via photo moderation

About the client

The client — a taxi aggregator — regularly checks the condition of the cars their drivers use, including vehicle model, branding, license plates, seat belts, cleanliness, damage, and more.

With more than 1.5 million cars and over USD 1.5 billion in revenue, the taxi app provides an essential and invaluable service to numerous customers on a daily basis.

To ensure customer safety and maintain brand image, the taxi aggregator checks the condition of the cars that taxi drivers use. Drivers submit photos of their vehicle via the app, including photos of the exterior and interior of their car, to ensure they’re compliant with the requirements before they are allowed to take orders.

Vehicle photos are reviewed and tagged for a comprehensive list of criteria, including photo quality, vehicle model and brand, license plate, cleanliness, damage (if any), seatbelts, and more.

The company has found that photo moderation and quality control directly correlate with the number of customer complaints. For example, when cars aren’t banned for severe damage, the number of complaints increases significantly, impacting customer retention rates.

The client uses two control methods for photo monitoring:

• Photo control of cars — Drivers take vehicle photos via the app and submit them to the service to check for compliance.

• Stationary control — A designated representative evaluates the car in person, fills out a questionnaire, and takes pictures. Then the team follows up and decides whether the car is compliant.

The challenge

The client’s goal was to decrease the number of user complaints by improving photo monitoring. They needed to label daily streams of image data with high accuracy.

To accommodate large volumes of data, the client was already using machine learning models to classify whether a car is compliant. However, these models weren’t always providing verdicts with a high level of confidence. For more complex cases (such as checking for cleanliness, damage, or branding), model accuracy was not meeting the service’s standards.

Key business metrics:

• The number of rides with customer complaints

• The number of customer complaints about the car's condition

Because photo moderation directly affects these metrics, the client prioritized accuracy and speed of labeling over cost.

Given the complex requirements for approving taxis, the client needed human input to ensure customer safety. In short, the client needed a quick way to assess 1.5 million cars for compliance with comfort and safety standards.

Solution

Toloka’s crowd of human labelers joined the photo moderation pipeline to boost accuracy. They played a crucial role in helping to classify complex cases by filling in the gaps in automatic classification and gathering data for model training and quality improvement.

The crowd provided verdicts for subjective criteria like cleanliness and damage, where ML models were performing with only 30% accuracy on average. Photos that the models labeled with a low confidence level were sent to Tolokers for evaluation. Tolokers also performed a quality check on the 1% of data that was labeled with high confidence by models.

The results

With the help of Toloka, the client could label larger volumes of data at a cost savings of about five times less per label compared to the alternative internal solution they were using.

At a significantly lower cost, Toloka's output quality remained high and met the client's expectations for complex classification cases. For relatively subjective evaluations, Toloker accuracy averaged 80-90% compared to 30% for machine learning models. For simple tasks, such as performing a license plate check, accuracy was close to 100%.

Toloka surpassed the client’s expectations for labeling speed, the ability to scale for large volumes, price per label, and output quality. The taxi app plans on using Toloka for other applications such as gathering data for future model training.