Toloka Research
Our mission
![](data:image/svg+xml,<svg display="block" role="presentation" viewBox="0 0 48 48" xmlns="http://www.w3.org/2000/svg"><path d="M 0 34 L 0 0 L 34 0 L 34 34 Z M 3 31 L 31 31 L 31 3 L 3 3 Z M 3 31 L 3 3 M 7.5 14.5 L 14.5 14.5 L 14.5 7.5 L 7.5 7.5 Z M 19.5 14.5 L 26.5 14.5 L 26.5 7.5 L 19.5 7.5 Z M 7.5 26.5 L 14.5 26.5 L 14.5 19.5 L 7.5 19.5 Z M 19.5 26.5 L 26.5 26.5 L 26.5 19.5 L 19.5 19.5 Z" fill="rgb(0, 0, 0)" height="34px" id="dHxwIblG9" transform="translate(7 7)" width="34px"/></svg>)
New data collection methods for SFT and RLHF that leverage synthetic data, AI feedback, and expert human-generated data.
![](data:image/svg+xml,<svg display="block" role="presentation" viewBox="0 0 48 48" xmlns="http://www.w3.org/2000/svg"><path d="M 4.3 35.7 C 2.933 34.1 1.875 32.308 1.125 30.325 C 0.375 28.342 0 26.233 0 24 C 0 19 1.75 14.758 5.25 11.275 C 8.75 7.792 13 6.05 18 6.05 C 18.333 6.05 18.675 6.058 19.025 6.075 C 19.375 6.092 19.683 6.117 19.95 6.15 L 15.9 2.1 L 18 0 L 26 8 L 18 16 L 15.85 13.85 L 20.45 9.25 C 20.083 9.183 19.65 9.133 19.15 9.1 C 18.65 9.067 18.267 9.05 18 9.05 C 13.833 9.05 10.292 10.5 7.375 13.4 C 4.458 16.3 3 19.833 3 24 C 3 25.7 3.283 27.4 3.85 29.1 C 4.417 30.8 5.283 32.283 6.45 33.55 Z M 16.1 35.5 C 16.1 34.8 15.85 34.083 15.35 33.35 C 14.85 32.617 14.308 31.867 13.725 31.1 C 13.142 30.333 12.6 29.525 12.1 28.675 C 11.6 27.825 11.35 26.933 11.35 26 C 11.35 24.167 12 22.6 13.3 21.3 C 14.6 20 16.167 19.35 18 19.35 C 19.833 19.35 21.4 20 22.7 21.3 C 24 22.6 24.65 24.167 24.65 26 C 24.65 26.933 24.4 27.825 23.9 28.675 C 23.4 29.525 22.858 30.333 22.275 31.1 C 21.692 31.867 21.15 32.617 20.65 33.35 C 20.15 34.083 19.9 34.8 19.9 35.5 Z M 16 40 L 16 37.5 L 20 37.5 L 20 40 Z M 31.7 35.7 L 29.6 33.6 C 30.6 32.4 31.417 30.983 32.05 29.35 C 32.683 27.717 33 25.933 33 24 C 33 21.8 32.542 19.708 31.625 17.725 C 30.708 15.742 29.333 14 27.5 12.5 L 29.7 10.3 C 31.633 11.967 33.167 13.975 34.3 16.325 C 35.433 18.675 36 21.233 36 24 C 36 26.233 35.625 28.342 34.875 30.325 C 34.125 32.308 33.067 34.1 31.7 35.7 Z" fill="rgb(0, 0, 0)" height="40px" id="w_1wCHMGy" transform="translate(6 2)" width="36px"/></svg>)
Improved approaches to model training and alignment
that enhance model capabilities in long-horizon reasoning and autonomous behavior.
![](data:image/svg+xml,<svg display="block" role="presentation" viewBox="0 0 48 48" xmlns="http://www.w3.org/2000/svg"><path d="M 0 34 L 0 0 L 34 0 L 34 34 Z M 3 31 L 31 31 L 31 3 L 3 3 Z M 7.731 26.5 L 10.731 26.5 L 10.731 17 L 7.731 17 Z M 23.27 26.5 L 26.269 26.5 L 26.269 7 L 23.27 7 Z M 15.5 26.5 L 18.5 26.5 L 18.5 21 L 15.5 21 Z M 15.5 17 L 18.5 17 L 18.5 13 L 15.5 13 Z M 3 31 L 3 3" fill="rgb(0, 0, 0)" height="34px" id="ftEu8fKCA" transform="translate(7 7)" width="34px"/></svg>)
High-quality evaluation metrics & benchmarks to measure performance in coding, math, reasoning, multilingualism, multimodality, and other complex tasks.
![](data:image/svg+xml,<svg display="block" role="presentation" viewBox="0 0 48 48" xmlns="http://www.w3.org/2000/svg"><path d="M 16 40 C 11.333 38.833 7.5 36.125 4.5 31.875 C 1.5 27.625 0 22.967 0 17.9 L 0 6 L 16 0 L 32 6 L 32 17.9 C 32 22.967 30.5 27.625 27.5 31.875 C 24.5 36.125 20.667 38.833 16 40 Z M 16 36.9 C 19.533 35.733 22.458 33.592 24.775 30.475 C 27.092 27.358 28.45 23.867 28.85 20 L 16 20 L 16 3.25 L 3 8.1 L 3 17.9 C 3 18.3 3.008 18.642 3.025 18.925 C 3.042 19.208 3.083 19.567 3.15 20 L 16 20 Z" fill="rgb(0, 0, 0)" height="40px" id="pNyMSpjWx" transform="translate(8 4)" width="32px"/></svg>)
Red-teaming methods for identifying model vulnerabilities and developing safety metrics such as harmfulness, security and CBRN risks, social bias, and more.
Our projects
BigCode: Open-scientific collaboration working on the responsible development of Large Language Models for Code
Publications
arXiv 2023
Conferences and events
We regularly hold tutorials and lead workshops at some of the biggest AI conferences around the globe.
Blog
Applied ML at Toloka
AI copilots
In-task tools help experts focus on quality: accurate fact checks, grammar checks, suggestions and more
Antifraud algorithms
Fraud prevention built into every data pipeline from start to finish to guarantee authentic human effort and expertise
Matching algorithms
Task distribution system matches tasks to the best qualified annotators and experts
Automated metrics
Our data quality metrics correlate with model performance gains for confidence in training data
Open job positions: