Teaching a 4B agent enterprise tool use: from 6% to 71% task success in under a day

Most tool-calling benchmarks measure whether a model can name the right function. They rarely measure whether it can hold a 30-turn conversation with a customer who changes their mind, work across dozens of similarly named APIs, follow company policy, and still land on the right outcome. That gap between benchmark skill and job skill is exactly what enterprises run into the moment they try to put an agent into production.

We wanted to see how far a small, off-the-shelf open model could close that gap, using nothing but our own data and a same-day training pipeline. The short version: a 4B model went from solving 6% of realistic enterprise support tasks to 71%, and the skill it picked up generalized to a business environment it never trained on and to public benchmarks it never saw.

The dataset: real enterprise tool use, built and owned by Toloka

The results rest on a private dataset of enterprise tool-use scenarios that we designed, built, and verified in house. It's not a repackaging of public benchmarks, and none of the models we evaluated had seen it before.

• Eleven simulated businesses, each with its own systems and policies: a retail brokerage support desk, a telecom operator, a travel marketplace, an airline cargo desk, a logistics provider, an internal corporate travel service, a quick-service restaurant chain, an airline, a pharmaceutical company, a bank's HR back office, and a manufacturer.

• Realistic systems as tools, not toy APIs: CRM, ticketing, billing, inventory, policy search, and more, dozens of endpoints per environment, with the naming overlaps and quirks of real enterprise stacks.

• A simulated customer who talks like a real one: natural language, changes of mind, withheld details, pushback.

• Verifiable outcomes. Every task has a defined end state. It only counts as solved if the right records changed in the right way and the customer was told the right thing. No judge opinions decide pass or fail.

• Long horizon. A typical task takes 10 to 15 tool calls across a multi-turn conversation, which is where wrong tools, wrong arguments, premature answers, and policy violations actually happen.

• Scale and hygiene. Roughly 1,900 tasks across all eleven environments, 374 of them held out for evaluation, and one full environment (airline cargo) kept entirely out of training as a zero-shot test.

That last point matters more than it might look. A held-out environment is the only way to tell whether a model learned the shape of enterprise tool use or just memorized eleven specific businesses.

Where a small open model starts

An off-the-shelf 4B open-weights model (Qwen3.5-4B) solves about 6% of these tasks on the first attempt, and 0% in four of the eleven environments.

That's far below what its published scores on popular tool-calling benchmarks would suggest. Our reading: small models are tuned hard against public leaderboards, and single-shot function-calling or short agentic benchmarks don't exercise what these scenarios demand: tracking a long conversation, choosing among dozens of similarly named tools, respecting company policy, and closing the loop with the customer. This is the gap the dataset is built to measure.

Step one: supervised fine-tuning on passing traces

We collected trajectories from several frontier models working the same tasks and kept only the ones that verifiably passed. We fine-tuned the base model on that mix with a light LoRA adapter: about 4,000 trajectories, two epochs, under two hours on an 8xH100 node.

The purpose of this stage is narrow: interface literacy. How to name tools, fill their arguments, sequence them, and answer the customer.

Result: the model solves 49.7% of evaluation tasks after SFT, up from 6.2% raw.

Step two: reinforcement learning against the environments

SFT teaches a model to imitate a trace. RL teaches it to finish the job. The agent runs each training task several times against a live copy of the environment with a simulated customer, is rewarded only for tasks it actually completes (verified by the environment, not a judge), and its policy is updated on the difference between its better and worse attempts, GRPO-style.

Weights update continuously while the model plays. The whole RL stage, 120 steps across 4 epochs, took well under a day of wall-clock time on 6 GPUs.

Measured over 374 evaluation tasks with 5 attempts each, paired against the SFT model's own attempts on the same tasks.

We also checked for reward hacking, since a model that games the pass/fail check is worse than useless in production. An independent rubric judge scored trajectories higher after RL (0.77 to 0.87), and the rate of "passed the check but violated the rubric" stayed at about 1% throughout training.

The held-out environment improved too. Airline cargo was never seen in training, no SFT traces, no RL rollouts. The SFT model solved 11.2% of its tasks; the RL model solves 43.5%, a jump of +32.3 percentage points (95% CI roughly ±4 pp). The agent learned how to work an enterprise system, not just the ten systems it trained on.

It transfers to public benchmarks

Training on our scenarios also improved the model on two public agent benchmarks it never saw, evaluated under the same conditions as the base model:

The BFCL gain holds across every group of the benchmark: single-turn, live, and multi-turn. On τ³, the model gains 6 to 11 points on airline and retail, and gives some back on telecom, where our scenarios taught it to reach for tools that belong to the τ³ customer instead of the agent.

We see both as evidence of transfer rather than memorization: the model isn't recalling a task list, it's picking up general tool discipline. That said, we'd call this a modest improvement for a model of this size, and we expect the next iteration to show a larger effect, more on that below.

What we take from this

Verifiable environments beat static data. SFT on frontier traces gets a small model to speak the interface. RL against a live environment teaches it to finish the job, and that generalizes to a business it never saw and to benchmarks it was never trained toward.

The dataset is the asset. Realistic systems, a demanding simulated customer, and verifiable outcomes are what make the reward trustworthy and the gains real. The same environments work as a benchmark, as an SFT source, and as an RL gym, which is why we license this data on an off-the-shelf basis and also run the full pipeline as a managed service for teams who'd rather hand us the environment and get back a tuned model.

It's fast once the infrastructure is in place. With environments, rollout workers, reward, and trainer wired into one continuous pipeline, the whole cycle, SFT, RL to 120 steps, evaluation, fits in a day on a handful of GPUs. Building that pipeline is the real investment; running it is cheap.

Where we expect the next numbers to move

We're treating this as a first data point, not a final one, and we want to be upfront about what's still in motion:

• Larger base models. A 4B model sits near the floor of what can hold a 30-turn enterprise conversation with dozens of tools in context. The same recipe on a larger base has more capacity to convert into general agentic skill, so we expect both the in-domain and transfer gains to grow. That's a hypothesis we intend to test, not a result we're claiming yet.

• A lighter SFT stage. The SFT stage imprints our dataset's style on the model, which likely caps how much of the RL gain carries over to public benchmarks. We're now testing a much lighter "bridge" SFT, just enough to teach the tool interface, so RL does more of the work and the model keeps more of its general ability. Early results are encouraging; we'll share details in a follow-up.

  
  

We'll update this post as those results land.