Toloka helps a smart device manufacturer test UX solutions — and branding concepts

Voice assistants, smart TVs, and other smart home features encompass a wide range of UX components that must be tested before hitting the market. That’s why a client of ours, a manufacturer of smart devices, recently reached out to us to help them collect feedback and compare design concepts for a better user experience.

Given that video and audio are the primary ways in which smart devices interact with users, we were able to leverage our strong capabilities, efficient pipeline, and the power of the crowd to help them navigate this process with notable success. Here’s how.

Applying three testing methods to find the optimal solution

The client applied three different methods for testing various UX solutions depending on their complexity: a UX lab, our Toloka platform, and A/B testing during production. To collect quantitative research data, the client leveraged the collective feedback of the crowd via Toloka as part of their UX testing strategy to identify user preferences and choose the best design concepts.

What were the results?

With the help of Toloka, the client was able to attain the data needed to evaluate key touch points and areas of improvement for each of their smart devices. Thanks to the feedback provided by the crowd, the client was able to make necessary changes to their products to provide a great user experience for their customers — and grow brand recognition. Keep reading to learn more about how Toloka helped the client assess their smart devices, including their TV, speaker, and home assistant, as well as their company logo.

Design testing for a smart TV startup sound

The client wanted to test their smart TV’s startup sound since they weren’t sure whether it was suitable. So, they decided to test it out via Toloka. The client asked the crowd the following questions:

How do you feel after hearing this sound? Positive, negative, neutral, interested, irritated, or other? Do you like the sound? Yes, no, or I don’t know. Rate the sound on a scale from 1 to 10. What comes to mind when you hear this sound? What does it sound like? For example, a sound that a TV makes when it’s turned on, music in a supermarket, or a soundtrack to a film.

After running these questions by Tolokers, the client found that the majority liked the sound and associated it with technology. The crowd found it to be appealing and interesting. The overall feedback indicated that this sound would be a good choice as part of the user experience.

Brand awareness testing for a smart TV remote control

Based on the assumption that a TV remote control is critical for brand recognition, the client wanted to test how well people could recognize different remote controls according to their brand.

The client’s design team asked the crowd if they could identify remote controls by leading smart TV manufacturers. Tolokers looked at images of TV remote controls from different companies and attempted to name the corresponding brand.

The results showed that only one was recognizable — Samsung’s, which indicated that a TV remote control isn’t critical for brand [AK1] recognition. Therefore, the design team was able to reset their design priorities.

Company logo visibility

When the client was still in their startup phase, they had a small logo as an icon on their mobile app’s voice assistant. As the company grew, the logo started appearing more frequently — on packaging, advertising, smart home devices, and mobile interfaces. The company went through a rebranding and the logo design no longer fit within the new brand identity. The client decided to update the logo and remove the gradient, which clashed with the brand’s new flat, minimalist design.

The client’s design team predicted that users wouldn’t notice the change, so they tested out their hypothesis by asking Tolokers to compare different logos to determine which one was genuine. The goal was to evaluate how memorable the original logo was. It turned out that their prediction was off the mark. The gradient actually played a significant role in the brand’s visibility on the market, as the crowd could attest to.

The client was concerned that a rapid redesign of the logo would negatively affect product metrics, so they divided the rebranding process into two iterations: first, they enlarged the triangle on the logo and then chose a new color — the intermediate between the gradient colors.

After deciding on the logo, the client ran several tests to see how the crowd liked it and whether they could distinguish the new logo from the old one and other “fake” logos. The good news is that the product metrics weren’t negatively affected, and since piloting the new logo, the client received only positive feedback from Tolokers.

UX testing for a smart speaker

The client wanted to enhance their smart speaker, which had a basic screen with a clock and simple animations showing the weather and a few emojis. To do this, the client compared three interfaces. The first one turned out to be childish and cartoonish and didn’t appeal to adults. It was hard for the crowd to read the emotions of the second one, and the winning design was retested.

The crowd was asked to respond to the following question: If you ask an assistant, “How are you doing?” and it shows you the following picture on the screen, what’s your impression? Tolokers recorded their answers in free form, including “happy”, “playful”, “flirty”, and so on. Pre-determined responses didn’t fit in this scenario because the idea was to gather genuine emotional responses. Some of the animations were changed after being run through the platform. For example, only the standard smiley face with heart-eyes was identified as depicting “love”, even though the design team viewed this emoji as too simple.

Design testing for smart home assistant sounds

When you ask your smart home assistant to turn something on it usually replies, “The feature is turned on.” The same goes for turning things off or when an error occurs. However, some users find this dialogue annoying. The client’s design team wanted to replace this saying with either a neutral jingle or a human-like reaction like “uh-huh.” However, when testing the latter, they discovered that people find it extremely unpleasant and even arrogant for a voice assistant to respond with something like “yeah, sure” to a homeowner’s request.

Therefore, the team moved on to neutral sounds. They asked Tolokers to find the most appealing sound among those produced by top competitors. Then they compared the sounds developed by the internal team with these audio recordings. If Tolokers preferred a particular sound over those of the competitors, it was selected. As a result, the design team was able to collect pleasant sounds for every scenario and add them to the assistant for a better user experience.

"It’s highly effective to be able to back up your design decisions with the collective opinion of the crowd, which also helps to prove design efficiency to the rest of the team. After receiving feedback from over 150 Tolokers, we could ensure that users would be satisfied with potential design alterations. The simplicity of Toloka and the quick response time allows us to test UX and branding hypotheses rapidly and make changes instantaneously."

– Sergey Kondaurov, Head of Smart Devices Design