DeveloperWeek 2023

Overcoming scaling challenges for complex systems with Toloka.

Toloka's crowdsourcing platform unites the individual efforts of millions of people around the world to label and process data for clients. The results are used in machine learning, research, software testing, technical support, content moderation, and other tasks.

Toloka's backend is a complex service for managing a huge "computing cluster", where people performing tasks act as nodes.

Artem will describe how we work out the problem of managing the "cluster" using the mechanism responsible for creating a list of available tasks for each Toloker (crowd worker).

He'll share three stories about developing various parts of the system. You'll also hear about the challenges we faced while scaling and how we solved them.