← Blog
/

Toloka Talks New York on making AI work for your business
Toloka Arena is live. See how your model ranks.
A model that tops a leaderboard can still stumble on the work a business needs done. That gap ran through Toloka Talks New York on October 28, where model builders and enterprise AI teams compared notes on what it takes to make AI work in production.
The panel brought together Shopify CTO Mikhail Parakhin and Liquid AI CEO Ramin Hasani, with Arnaud Stiegler, who works on agentic coding capabilities at Reflection AI, sharing a researcher's perspective on building models that perform beyond the benchmark. Toloka CEO Olga Megorskaya hosted the evening.
Start with your own test
Public benchmarks give labs a shared target. A business has a narrower question: how will this model handle the work waiting for it?
Shopify answers that with its own history. Mikhail's team draws on thousands of code changes that caused production incidents over several years, then checks whether a model can catch the problems that slipped through human review. The team also counts false alarms, because engineers wouldn't get much done if they had to investigate every harmless change.
Liquid AI comes at it from the other side of the table. When adapting models for customers, Ramin's team takes its lead from the customer's own internal evaluations. As Arnaud pointed out, a researcher is unlikely to come up with a helpful test without knowing the details of a customer's work.
The test shapes the model
A test is only as good as what it checks, and the gaps show up later. Arnaud described training tasks that asked a model to write a draft before completing the work. Nobody had included a check for the draft, so the model could skip that step and still earn credit. After training, users found it ignored parts of their requests.
Training data has the same weak spot. In one synthetic dataset his team generated, 90% of the task descriptions began with "could you," because a control meant to encourage variety had been left out. Drawing on real material, including enterprise data, helps. So does deliberately varying the requests and reviewing how models attempt the work, which can expose a faulty environment or a way to earn the reward without doing what was asked.
"You should know what good looks like," Arnaud said. At the scale his team works, finding a problem only after training is an expensive mistake.
Picking the right-sized model
Once a company knows what good looks like, it can match the model to the job, and the best fit isn't always the biggest one. Ramin and Mikhail both talked about companies taking more control of their AI: Ramin by giving enterprises the tools to adapt models themselves, Mikhail by running models within Shopify's own infrastructure.
Neither treated it as all-or-nothing. Mikhail expected large closed models to stay useful for demanding coding work. For repetitive tasks, a smaller model is an easier case to make.
Shopify's search shows why. A Liquid AI model rewrites shoppers' queries into a form the search system can use, in around 30 milliseconds. A round trip to a remotely hosted model could take too long, and that's before counting the cost at Shopify's scale.
That raised a question from the audience: is fine-tuning worth the effort when new models arrive so often? Mikhail's answer was that once the data and training process are in place, the savings on repeatable work can justify fine-tuning again with each new model.
Production as a training ground
The work doesn't stop at deployment. Shopify uses a grader to flag conversations that could have gone better, and several larger models then attempt improved responses. Where the models fall short, human experts provide better examples through Toloka, and those feed into further training.
Keeping the bar steady
Through Toloka's work with both labs and enterprises, Olga sees the gap the panel kept returning to. Labs often work toward a familiar set of public benchmarks, while a business needs to know how a model will handle its own work. Shopify Sidekick shows what closing that gap looks like: its evaluation data is updated weekly to reflect current use.
The examples change, Olga noted, but the standards for judging good work don't have to change with them. Setting those standards takes domain experts, the people who deal with the tasks every day and know where models tend to fail.
Whether you're training frontier models or putting them to work in your business, connect with our team to learn how Toloka can support your model training and evaluation efforts.
Thanks to everyone who joined us in New York, and to our panelists for sharing their experiences. Browse photos from the evening below.
Subscribe to our newsletter
Product updates, case studies and the latest news from our team, straight to your inbox









