Production‑grade agent data
Toloka builds environments & RL-gyms, collecting trajectories and graded eval signals to train and evaluate AI agents. Get the data you need without diverting your researchers into data ops
Trusted by Leading AI Teams
What we deliver
We collaborate with your team to define robust success criteria, then engineer reproducible data and environments that integrate with your training and evaluation workflows.
Virtual environments
Human-simulated virtual companies
Computer-use mockups
Synthetic companies
MCU mockups
Agent capability evaluation
MCP-bench extentions
TinyTAU for on-device agents
TAU-bench extentions
Agent trajectory data
Coding agent safety evaluation
MCP injection vulnerability assessment
Computer-use agent injection vulnerability red-teaming
Agent safety data
Trajectory demonstrations
Trajectory evaluetions
Expertise domains
Enterprise systems
Salesforce
Servicenow
Zendesk
Software engineering
Pyton
Javascript
C++
Typescript
Java
Rust
Golang
Quantitative sciences
Mathematics
Physics
Chemistry
Data analysis
Agent types we work for
Corporate assistants
Automate tasks and workflows by interacting with internal tools, knowledge bases, and policies to enhance employee productivity (e.g., customer support, sales, marketing, recruitment, etc.)
OS agents
Manage interactions with operating systems and mobile devices, including smartphones and wearables
How it works:
a managed pipeline
built by engineers,
for engineers
Managed, end-to-end data operations
You provide objectives, guidelines, and constraints. We design the environment, run data collection, generation, and annotation at scale, then return versioned datasets, eval reports, and deliverables ready for training.
Automated QA
Tool-enabled checks for rubric adherence, logical consistency across steps, environment invariants, and task completion.
Structural validation of traces (schema, required fields, value ranges).
Signals produced
Per-step and per-task labels (guideline adherence, failure categories, safety flags).
Calibrated scores for SFT selection and RLAIF reward shaping.
Senior human review
Senior reviewers audit complex or flagged trajectories and a statistically sound sample of the rest.
Human feedback is used to finetune the QA Agent between batches to reduce drift and improve recall on rare errors.
Task execution
Human experts complete the task; we log the raw trace.
Privacy, security, and reproducibility
![](data:image/svg+xml,<svg display="block" role="presentation" viewBox="0 0 48 48" xmlns="http://www.w3.org/2000/svg"><path d="M 15.5 15 C 16.472 15 17.298 14.66 17.979 13.979 C 18.66 13.298 19 12.472 19 11.5 C 19 10.528 18.66 9.702 17.979 9.021 C 17.298 8.34 16.472 8 15.5 8 C 14.528 8 13.702 8.34 13.021 9.021 C 12.34 9.702 12 10.528 12 11.5 C 12 12.472 12.34 13.298 13.021 13.979 C 13.702 14.66 14.528 15 15.5 15 Z M 15.519 23 C 16.827 23 18.068 22.738 19.244 22.214 C 20.42 21.689 21.459 20.94 22.361 19.966 C 21.259 19.314 20.133 18.824 18.984 18.495 C 17.836 18.165 16.681 18 15.519 18 C 14.242 18 13.013 18.168 11.832 18.504 C 10.652 18.84 9.594 19.327 8.658 19.966 C 9.548 20.94 10.583 21.689 11.765 22.214 C 12.947 22.738 14.199 23 15.519 23 Z M 35.277 37.404 L 25.319 27.446 C 23.901 28.577 22.39 29.452 20.785 30.071 C 19.18 30.69 17.418 31 15.5 31 C 11.174 31 7.51 29.498 4.506 26.494 C 1.502 23.49 0 19.826 0 15.5 C 0 11.174 1.502 7.51 4.506 4.506 C 7.51 1.502 11.174 0 15.5 0 C 19.826 0 23.49 1.502 26.494 4.506 C 29.498 7.51 31 11.174 31 15.5 C 31 17.418 30.69 19.183 30.071 20.794 C 29.452 22.406 28.577 23.921 27.446 25.338 L 37.385 35.277 Z M 15.5 28 C 18.962 28 21.91 26.782 24.346 24.346 C 26.782 21.91 28 18.962 28 15.5 C 28 12.038 26.782 9.09 24.346 6.654 C 21.91 4.218 18.962 3 15.5 3 C 12.038 3 9.09 4.218 6.654 6.654 C 4.218 9.09 3 12.038 3 15.5 C 3 18.962 4.218 21.91 6.654 24.346 C 9.09 26.782 12.038 28 15.5 28 Z M 15.5 15.5" fill="rgb(0, 0, 0)" height="37.403999999999996px" id="WdRhxOeMe" transform="translate(6.5 4.5)" width="37.3845px"/></svg>)
PII scrubbing, policy-compliant use of foundation models, and client-approved data handling.
![](data:image/svg+xml,<svg display="block" role="presentation" viewBox="0 0 48 48" xmlns="http://www.w3.org/2000/svg"><path d="M 16 20.05 M 16 40 C 11.333 38.833 7.5 36.125 4.5 31.875 C 1.5 27.625 0 22.967 0 17.9 L 0 6 L 16 0 L 32 6 L 32 17.9 C 32 18.333 31.992 18.767 31.975 19.2 C 31.958 19.633 31.917 20.067 31.85 20.5 C 31.683 20.5 31.542 20.492 31.425 20.475 C 31.308 20.458 31.167 20.45 31 20.45 C 30.633 20.45 30.267 20.475 29.9 20.525 C 29.533 20.575 29.167 20.633 28.8 20.7 C 28.867 20.233 28.917 19.767 28.95 19.3 C 28.983 18.833 29 18.367 29 17.9 L 29 8.1 L 16 3.2 L 3 8.1 L 3 17.9 C 3 22.267 4.208 26.208 6.625 29.725 C 9.042 33.242 12.167 35.633 16 36.9 C 16.9 36.6 17.775 36.217 18.625 35.75 C 19.475 35.283 20.283 34.767 21.05 34.2 L 21.05 37.85 C 20.283 38.317 19.475 38.733 18.625 39.1 C 17.775 39.467 16.9 39.767 16 40 Z M 23.65 40.05 L 23.65 29.75 L 25.75 29.75 L 25.75 27.75 C 25.75 26.55 26.142 25.533 26.925 24.7 C 27.708 23.867 28.683 23.45 29.85 23.45 C 31.017 23.45 31.983 23.867 32.75 24.7 C 33.517 25.533 33.9 26.55 33.9 27.75 L 33.9 29.75 L 36 29.75 L 36 40.05 Z M 27.75 29.75 L 31.95 29.75 L 31.95 27.75 C 31.95 27.083 31.758 26.533 31.375 26.1 C 30.992 25.667 30.483 25.45 29.85 25.45 C 29.183 25.45 28.667 25.667 28.3 26.1 C 27.933 26.533 27.75 27.083 27.75 27.75 Z" fill="rgb(0, 0, 0)" height="40.049993133544945px" id="Hho9oIXBB" transform="translate(8 3.95)" width="36.00000915527346px"/></svg>)
Secure, containerized environments and controlled credentials in testbeds.
![](data:image/svg+xml,<svg display="block" role="presentation" viewBox="0 0 48 48" xmlns="http://www.w3.org/2000/svg"><path d="M 1.1 11.65 L 1.1 1.25 L 4.1 1.25 L 4.1 6.55 C 5.833 4.517 7.892 2.917 10.275 1.75 C 12.658 0.583 15.183 0 17.85 0 C 22.883 0 27.167 1.733 30.7 5.2 C 34.233 8.667 36 12.9 36 17.9 L 36 18.4 L 33 18.4 L 33 17.75 C 33 13.617 31.517 10.125 28.55 7.275 C 25.583 4.425 22.017 3 17.85 3 C 15.583 3 13.458 3.517 11.475 4.55 C 9.492 5.583 7.767 6.95 6.3 8.65 L 11.55 8.65 L 11.55 11.65 Z M 0 17.95 L 3 17.95 C 3 22.117 4.433 25.667 7.3 28.6 C 10.167 31.533 13.683 33 17.85 33 L 18.45 33 L 20.15 35.85 C 19.783 35.917 19.408 35.958 19.025 35.975 C 18.642 35.992 18.25 36 17.85 36 C 12.883 36 8.667 34.242 5.2 30.725 C 1.733 27.208 0 22.95 0 17.95 Z M 22 23.9 L 16.55 18.55 L 16.55 7.85 L 19.55 7.85 L 19.55 17.3 L 23.6 21.25 Z M 30.75 41.5 L 30.25 38.2 C 29.65 38.033 29.083 37.808 28.55 37.525 C 28.017 37.242 27.517 36.867 27.05 36.4 L 24.3 37 L 23.05 34.9 L 25.4 32.7 C 25.3 32.3 25.25 31.883 25.25 31.45 C 25.25 31.017 25.3 30.6 25.4 30.2 L 23.05 28 L 24.3 25.9 L 27.05 26.5 C 27.517 26.033 28.017 25.658 28.55 25.375 C 29.083 25.092 29.65 24.867 30.25 24.7 L 30.75 21.4 L 33.45 21.4 L 33.95 24.7 C 34.55 24.867 35.117 25.092 35.65 25.375 C 36.183 25.658 36.683 26.033 37.15 26.5 L 39.9 25.9 L 41.15 28 L 38.8 30.2 C 38.9 30.6 38.95 31.017 38.95 31.45 C 38.95 31.883 38.9 32.3 38.8 32.7 L 41.15 34.9 L 39.9 37 L 37.15 36.4 C 36.683 36.867 36.183 37.242 35.65 37.525 C 35.117 37.808 34.55 38.033 33.95 38.2 L 33.45 41.5 Z M 32.1 35.45 C 33.3 35.45 34.267 35.083 35 34.35 C 35.733 33.617 36.1 32.65 36.1 31.45 C 36.1 30.25 35.733 29.283 35 28.55 C 34.267 27.817 33.3 27.45 32.1 27.45 C 30.9 27.45 29.933 27.817 29.2 28.55 C 28.467 29.283 28.1 30.25 28.1 31.45 C 28.1 32.65 28.467 33.617 29.2 34.35 C 29.933 35.083 30.9 35.45 32.1 35.45 Z" fill="rgb(0, 0, 0)" height="41.50000152587893px" id="n9Lx8CTzr" transform="translate(6 6)" width="41.149995422363304px"/></svg>)
Versioned environments, deterministic resets, and audit logs for exact repro.
Partner with Toloka
![](data:image/svg+xml,<svg display="block" role="presentation" viewBox="0 0 48 48" xmlns="http://www.w3.org/2000/svg"><path d="M 20 40 C 17.233 40 14.633 39.475 12.2 38.425 C 9.767 37.375 7.65 35.95 5.85 34.15 C 4.05 32.35 2.625 30.233 1.575 27.8 C 0.525 25.367 0 22.767 0 20 C 0 17.233 0.525 14.633 1.575 12.2 C 2.625 9.767 4.05 7.65 5.85 5.85 C 7.65 4.05 9.767 2.625 12.2 1.575 C 14.633 0.525 17.233 0 20 0 C 22.767 0 25.367 0.525 27.8 1.575 C 30.233 2.625 32.35 4.05 34.15 5.85 C 35.95 7.65 37.375 9.767 38.425 12.2 C 39.475 14.633 40 17.233 40 20 C 40 22.767 39.475 25.367 38.425 27.8 C 37.375 30.233 35.95 32.35 34.15 34.15 C 32.35 35.95 30.233 37.375 27.8 38.425 C 25.367 39.475 22.767 40 20 40 Z M 20 37 C 24.746 37 28.766 35.353 32.06 32.059 C 35.353 28.766 37 24.746 37 20 C 37 15.254 35.353 11.234 32.06 7.941 C 28.766 4.647 24.746 3 20 3 C 15.254 3 11.234 4.647 7.94 7.941 C 4.647 11.234 3 15.254 3 20 C 3 24.746 4.647 28.766 7.94 32.059 C 11.234 35.353 15.254 37 20 37 Z M 20 32 C 16.667 32 13.833 30.833 11.5 28.5 C 9.167 26.167 8 23.333 8 20 C 8 16.667 9.167 13.833 11.5 11.5 C 13.833 9.167 16.667 8 20 8 C 23.333 8 26.167 9.167 28.5 11.5 C 30.833 13.833 32 16.667 32 20 C 32 23.333 30.833 26.167 28.5 28.5 C 26.167 30.833 23.333 32 20 32 Z M 20 29 C 22.5 29 24.625 28.125 26.375 26.375 C 28.125 24.625 29 22.5 29 20 C 29 17.5 28.125 15.375 26.375 13.625 C 24.625 11.875 22.5 11 20 11 C 17.5 11 15.375 11.875 13.625 13.625 C 11.875 15.375 11 17.5 11 20 C 11 22.5 11.875 24.625 13.625 26.375 C 15.375 28.125 17.5 29 20 29 Z M 20 24 C 18.9 24 17.958 23.608 17.175 22.825 C 16.392 22.042 16 21.1 16 20 C 16 18.9 16.392 17.958 17.175 17.175 C 17.958 16.392 18.9 16 20 16 C 21.1 16 22.042 16.392 22.825 17.175 C 23.608 17.958 24 18.9 24 20 C 24 21.1 23.608 22.042 22.825 22.825 C 22.042 23.608 21.1 24 20 24 Z" fill="rgb(0, 0, 0)" height="40px" id="WWfP_ajs5" transform="translate(4 4)" width="40px"/></svg>)
Keep your research org focused on model innovation; offload environment engineering, data collection, and QA operations to a team that does this full‑time.
![](data:image/svg+xml,<svg display="block" role="presentation" viewBox="0 0 48 48" xmlns="http://www.w3.org/2000/svg"><path d="M 16.35 23.7 C 17.117 24.467 18.225 24.825 19.675 24.775 C 21.125 24.725 22.183 24.183 22.85 23.15 L 33.65 6.2 L 16.9 17.15 C 15.9 17.817 15.367 18.883 15.3 20.35 C 15.233 21.817 15.583 22.933 16.35 23.7 Z M 19.9 0 C 21.8 0 23.783 0.308 25.85 0.925 C 27.917 1.542 29.9 2.6 31.8 4.1 L 29.2 5.95 C 27.7 4.95 26.092 4.208 24.375 3.725 C 22.658 3.242 21.166 3 19.899 3 C 15.217 3 11.23 4.67 7.938 8.011 C 4.646 11.351 3 15.397 3 20.149 C 3 21.65 3.208 23.167 3.625 24.7 C 4.042 26.233 4.633 27.65 5.4 28.95 L 34.35 28.95 C 35.083 27.75 35.667 26.35 36.1 24.75 C 36.533 23.15 36.75 21.583 36.75 20.05 C 36.75 18.65 36.542 17.142 36.125 15.525 C 35.708 13.908 34.967 12.417 33.9 11.05 L 35.85 8.45 C 37.117 10.317 38.067 12.192 38.7 14.075 C 39.333 15.958 39.683 17.85 39.75 19.75 C 39.817 21.75 39.617 23.633 39.15 25.4 C 38.683 27.167 38 28.8 37.1 30.3 C 36.7 31.067 36.275 31.533 35.825 31.7 C 35.375 31.867 34.817 31.95 34.15 31.95 L 5.6 31.95 C 5.033 31.95 4.475 31.808 3.925 31.525 C 3.375 31.242 2.967 30.833 2.7 30.3 C 1.833 28.7 1.167 27.075 0.7 25.425 C 0.233 23.775 0 22.017 0 20.15 C 0 17.383 0.525 14.775 1.575 12.325 C 2.625 9.875 4.05 7.742 5.85 5.925 C 7.65 4.108 9.761 2.667 12.184 1.6 C 14.607 0.533 17.179 0 19.9 0 Z M 19.45 16.55" fill="rgb(0, 0, 0)" height="31.94999999999999px" id="vRKonqjAd" transform="translate(4 8.05)" width="39.76250000000001px"/></svg>)
Faster to first useful dataset; more flexible than hiring for bursty, specialized work.
![](data:image/svg+xml,<svg display="block" role="presentation" viewBox="0 0 48 48" xmlns="http://www.w3.org/2000/svg"><path d="M 18 35.6 L 0 21.6 L 2.5 19.75 L 18 31.8 L 33.5 19.75 L 36 21.6 Z M 18 28 L 0 14 L 18 0 L 36 14 Z M 18 12.95 M 18 24.2 L 31.1 14 L 18 3.8 L 4.9 14 Z" fill="rgb(0, 0, 0)" height="35.60000000000002px" id="qyo_7ofoU" transform="translate(6 5.9)" width="36px"/></svg>)
Depth in agentic data: instrumented, stateful environments—not just annotation.
![](data:image/svg+xml,<svg display="block" role="presentation" viewBox="0 0 48 48" xmlns="http://www.w3.org/2000/svg"><path d="M 14.95 25.95 L 29.4 11.5 L 27.25 9.35 L 14.95 21.65 L 9 15.7 L 6.85 17.85 Z M 0 36 L 0 0 L 36 0 L 36 36 Z M 3 33 L 33 33 L 33 3 L 3 3 Z M 3 33 L 3 3" fill="rgb(0, 0, 0)" height="36px" id="JuOhQ7BU3" transform="translate(6 6)" width="36px"/></svg>)
Hybrid QA that blends tool‑enabled checks with senior human judgment, tuned to your rubric.
![](data:image/svg+xml,<svg display="block" role="presentation" viewBox="0 0 48 48" xmlns="http://www.w3.org/2000/svg"><path d="M 21.95 36 L 7.45 28.05 L 7.45 16.05 L 0 12 L 21.95 0 L 44 12 L 44 27.85 L 41 27.85 L 41 13.75 L 36.45 16.05 L 36.45 28.05 Z M 21.95 20.6 L 37.7 12 L 21.95 3.55 L 6.3 12 Z M 21.95 32.6 L 33.45 26.25 L 33.45 17.85 L 21.95 24 L 10.45 17.75 L 10.45 26.25 Z M 22 20.6 M 21.95 24.3 M 21.95 24.3" fill="rgb(0, 0, 0)" height="36px" id="Re_fEKe_o" transform="translate(2 6)" width="44px"/></svg>)
A rigorously vetted expert network with measurable quality controls.
![](data:image/svg+xml,<svg display="block" role="presentation" viewBox="0 0 48 48" xmlns="http://www.w3.org/2000/svg"><path d="M 33.8 36 L 20.65 22.85 C 19.65 23.717 18.484 24.392 17.152 24.875 C 15.82 25.358 14.403 25.6 12.9 25.6 C 9.295 25.6 6.243 24.35 3.746 21.85 C 1.249 19.35 0 16.333 0 12.8 C 0 9.267 1.25 6.25 3.75 3.75 C 6.25 1.25 9.275 0 12.825 0 C 16.375 0 19.392 1.25 21.875 3.75 C 24.358 6.25 25.6 9.269 25.6 12.807 C 25.6 14.236 25.367 15.617 24.9 16.95 C 24.433 18.283 23.733 19.533 22.8 20.7 L 36 33.8 Z M 12.85 22.6 C 15.558 22.6 17.861 21.642 19.757 19.725 C 21.652 17.808 22.6 15.5 22.6 12.8 C 22.6 10.1 21.652 7.792 19.757 5.875 C 17.861 3.958 15.558 3 12.85 3 C 10.114 3 7.788 3.958 5.873 5.875 C 3.958 7.792 3 10.1 3 12.8 C 3 15.5 3.958 17.808 5.873 19.725 C 7.788 21.642 10.114 22.6 12.85 22.6 Z" fill="rgb(0, 0, 0)" height="36px" id="Gcpv4RZzh" transform="translate(6 5.95)" width="36px"/></svg>)
Active R&D posture; we collaborate on novel evals and safety protocols with leading labs.
Learn more about Toloka
Trusted by Leading AI Teams