Improving translation quality

About the client

Machine translation services have made significant progress in recent years, becoming more accurate and truer to content, context, and meaning with each new advancement.

Our client provides automatic translations in 100 supported languages to over 60 million monthly users. The service automatically translates words, expressions, and text across images, videos, websites, and mobile apps using machine translation technology.

The challenge

To enhance translation quality and identify areas for improvement, the team routinely pushes new models into production and compares the results with current models and competitor output.

The client’s other key business goals include:

• User retention and engagement growth

• Revenue growth from B2B sales

• Growth of other services (for example, in-browser translation)

Humans are still more fluent than machines at evaluating translation quality. That's why human labeling is essential for comparing machine translation output and finding and fixing translation errors.

Solution

Toloka provided the client with quality assessments carried out and verified by humans rather than machines. In short, the crowd was asked to compare which translation was better: 1) the current one in production, 2) the one that was newly created, or 3) the competitor’s version. They were also tasked with finding and correcting any errors.

Tolokers were given two text units (either a word, phrase, sentence, or paragraph) with the aim of:

• Comparing two translations to choose the best one

• Rating translation quality on a scale of 0 to 100

• Detecting errors in translations

• Validating sentence segmentation after translation

Results

The numbers improved across the board, including the retention of weekly and monthly active users. Other internal metrics based on labeling results also improved with human input from the Toloka crowd.

Toloka’s solution was cost-efficient — the average price per label for pairwise quality assessment was five times lower than the alternative labeling solution used by the client (~$0.02 vs. $0.1).

Tolokers spent the same amount of time on each label as the client’s internal annotators (about 1.3 minutes per label), but the average project was completed much faster in Toloka because of the large crowd of skilled Tolokers available to evaluate translations.

Toloka’s fast and reliable solution helped support ongoing improvements to the accuracy of machine translation models to give the client a competitive edge.