Object detection and matching, image-to-text, and photo moderation for geoservices

About the client

The client, a geo services company, maintains digital maps used for navigation in multiple apps where customers look for directions and traffic information, plan routes, and search for places and businesses. The maps must be continually updated to reflect real-life changes in buildings, roads, and other outdoor sites.

The client routinely focuses on these goals:

• Improving user experience

• Increasing map coverage

• Boosting the accuracy of displayed objects

• Enhancing details in current objects and adding new ones

• Automating updates of cartographic information

• Expanding into new regions

Object detection and matching, image-to-text, and photo moderation are all equally important in addressing these areas.

For example, the client collects datasets of photos of areas on the map. They use automatic object detection to label road signs, traffic lights, and address plates. Then they verify the address plates by checking that the same object is depicted in two photos or that a road sign that was visible in one photo is also visible in another.

The model results aren’t always accurate and there is a persistent need to validate objects in new images. In addition, other types of tasks arise sporadically — for example, detecting road signs in a new country or detecting an object that was not detected prior.

Challenge

While machine learning models can classify large volumes of data, they don’t label all objects with a high level of confidence. Human input is needed to validate model output and to prepare training data for these models.

Improving map coverage and accuracy requires data labelers to accurately detect various objects (such as traffic lights, road signs, house numbers, and more), retrieve data from photos (like deciphering house numbers), match the same objects in two different photos, and evaluate the quality of the data being labeled. Humans still perform better than machines at all of these tasks.

When the ML model detects something on a photo that needs to be updated on the map, it flags the item for review by a professional cartographer, who approves the change. Customers of the service can also submit corrections for review, and these suggestions need to be moderated. The client’s key business metrics for object detection include the percentage of actual changes made to a map in response to the model’s suggestions and in response to user suggestions.

The client needed human labelers to validate model output, check for inconsistencies between the results and current map displays, and verify whether an object is actually shown in a photo. They also needed human labelers to collect datasets and moderate input data for model training.

Solution

Toloka provided object detection and matching services along with image-to-text and photo moderation capabilities. Our crowd contributors helped validate model results for images labeled with low confidence, build datasets for model training, and moderate source data for photo quality and suitability. Their tasks included:Outlining objects in photos:

• Checking object visibility in photos

• Comparing objects in different photos

• Matching objects in two different photos

• Classifying photos

• Marking house numbers, traffic lights, and road signs using bounding boxes

• Decoding house numbers (image-to-text)

• Checking that houses are correctly identified and outlined in satellite images

• Evaluating image quality (discarding blurry or unusable images)

Results

Human data labeling with Toloka was instrumental in helping the client validate model output, build training datasets, and moderate source data.

All key metrics improved, including the percentage of map changes approved by cartographers in relation to the number of suggestions made by the model and by users.

When it comes to geo services, machine learning models are only able to produce a small percentage of positive verdicts that generally do not correspond to the current map. For map updates suggested by the model, cartographers usually confirm only 1-2% of verdicts or hypotheses as being correct due to the presence of many false positives. Toloka played a key role in helping the client screen out these false positives and improve the quality of the model by providing additional labeling.

After the Toloka crowd filtered out which suggestions should be sent to cartographers, the percentage of approved verdicts rose significantly to nearly 50%, making map updates considerably more efficient.