Post-training for bank support: the agent found the recipe, the data set the limit

Getting a model to follow bank policy takes its fair share of work. Even after dozens of training experiments, an engineer might keep wondering whether to change the training method or give the model more examples of the situations where it keeps getting wrong.

In Toloka's bank-support use case, there was reason to do both. An autoresearch agent trained 62 versions of a small model, taking it from solving 1% of unseen synthetic conversations to 93.8%. Along the way, it compared full fine-tuning with LoRA, varied the training data, added reinforcement learning and kept the model's general skills intact.

However, it couldn't fix the dataset it was given. Some of the remaining mistakes involved situations that the training data barely covered. Automated experimentation gives engineers far more evidence without managing every training run themselves. Deciding what to do with that evidence, including whether the evaluation reflects the task and what the training examples are missing, still calls for judgment.

Bank policy leaves little room for a nearly right answer

Handling one refund is straightforward enough. But what happens when a customer asks about two, then only provides the details for one? In our setup, the bank's policy is explicit: when a customer raises two or more issues, every reply has to keep all of them in the chat until they are resolved.

In the study, models would sometimes explain what to do about the first refund and leave the second unresolved. An otherwise reasonable reply would fail because the other refund still needed a next step, even if it was simply a question asking for more information. Omitting an unresolved issue was the most common first policy error in the test, even among frontier models.

When it came to Qwen 3.5 4B, the assignment involved handling two customer issues over three to five turns. Doing so meant adhering to about 1,000 words of bank policy, following seven style rules and keeping every reply within 110 words. Refunds and transfers were part of the policy, alongside card security, scams, and existing support cases. Details revealed later in a conversation could change what the model needed to do.

Every reply in a conversation had to pass both the policy and style checks in order to be counted as solved. Before post-training, the 4B model solved just one of the 96 unseen conversations.

Putting the agent to work

The autoresearch agent ran the experiments, covering full fine-tuning and LoRA, different amounts of training data and further runs testing reinforcement learning.

Training data came from a controlled synthetic pipeline. After code generated cases and answer keys, language models wrote the customer messages and teacher replies. A judge model then checked the replies against bank policy, while code looked at the style rules. Keeping only conversations in which every teacher reply passed produced a training set of 2,048 conversations.

Alongside the main evaluation, another 96 conversations tested what happened when a customer changed an important fact. In half of the cases, the new information changed the correct action. Training approaches with similar main scores would turn out to perform differently on the changed-fact test.

What the experiments found

Full fine-tuning, 2 epochs, best of the learning rates tested.

Every increase in the training set brought an improvement, which is what you'd expect. The more telling result is how the agent used that scale. It tested different approaches on the smaller data sets, kept the most promising ones and scaled only those up. Trained on the full 2,048 conversations, the preselected recipes gained more than any change to the training technique alone.

Choosing between full fine-tuning and LoRA was less straightforward. Full fine-tuning came out slightly ahead on the main test, scoring 92.7% against LoRA's 91.7%. But when customers changed a fact, LoRA solved 79.2% of conversations compared with 63.5% for full fine-tuning.

Adding reinforcement learning after full fine-tuning addressed much of the weakness on the changed-fact eval dataset. The changed-fact score rose to 88.5%, while the main score reached 93.8%. Of the 96 main-test conversations, the final model solved 90. None failed on style, while even top proprietary models did.

Preserving general instruction-following ability took some attention too. Early versions lost some of it, with IFEval scores falling as low as 46.8% from the untuned model's 81.5%. During supervised training, the team replayed the model's own answers to general prompts, which accounted for half the training tokens. The final IFEval score matched the untuned model's 81.5%.

Checking whether post-training works for your task

The study also shows how little it takes to answer a practical question: is post-training worth pursuing for this task at all?

A single training run took about an hour on one H100 GPU:

LoRA fits on smaller GPUs, or several runs can share one H100.

The early signal arrives quickly too. As the scaling results show, a few hundred synthetic conversations are enough to judge whether a small model can learn the task, before anyone commits to a full project.

A smaller option for bank support

Main evaluation set, 96 conversations. The post-trained model was trained with the engineered prompt, so it has no default-prompt score.

Gemini 3.8 Flash and GLM-5.3, two widely used frontier models, both ran with reasoning switched on. Under the benchmark assumptions, the small model was 22 times faster than Gemini at roughly one fifty-eighth of the serving cost.

Could a better prompt get a frontier model further? In Gemini's case, reminding the model to address every unresolved issue puts it two solved conversations ahead of the post-trained model. GLM gained three. 

Where the agent hit the limit of the data

The agent found a strong recipe. It couldn't find examples that weren't in the data.

Some of the remaining errors trace back to underrepresented situations such as the four-day transfers. Passing the quality checks hadn't given every policy decision enough coverage.

Domain experts can contribute directly by reviewing failed replies and correcting the reasoning behind them. Writing conversations around the relevant policy decisions would give the model more opportunities to learn. Checking the answer keys and the judge's verdicts would also help the team reward behavior that would be acceptable in practice.

Bringing in real support traffic

To decide which missing examples deserve attention first, the team would need production conversations. Synthetic data was used throughout the study, so real support traffic would help establish how often particular situations occur. It could also introduce wording or combinations of issues the generator missed. A separate evaluation drawn from real traffic would then test whether the improvements carry over.

Expert data work has a place in Toloka's autoresearch process for precisely that reason. Alongside a trained model, the experiments leave a record of failures for engineers and domain experts to examine. After correcting or adding examples, they can run another experiment to see what improves. In bank support, the four-day transfer cases offer a clear place to start.

Benchmark details

Both the main evaluation and the changed-fact evaluation contained 96 synthetic conversations. The study was a demonstration, not a live bank deployment.

Latency was measured one request at a time over eight conversations comprising 27 replies. For the 4B serving-cost estimate, the team used measured throughput with 16 conversations running in parallel on one H100 GPU, assuming $4 per GPU-hour. Frontier-model costs came from the API gateway. Development and training costs aren't included.