Production LLM workloads tend to run into one of two cost problems, one of which is model cost. A frontier API may work well for a task that barely changes — parsing documents, triaging tickets, or classifying records — but you still pay frontier-model prices for a narrow slice of its capability. The quality may be fine; the cost isn't.

The other is prompt cost. A production prompt can grow to a few thousand tokens of instructions, rules, and examples. That static text is processed on every request and re-processed millions of times a month before the model reaches the actual input.

Today we're launching two early-access tools on the Toloka Platform to address those costs. Each takes a JSONL file and a budget. Neither requires an ML research team, a cluster, or a YAML file.

The tools have different inputs and workflows, so we'll explain them separately.

Fine-tuning: keep the quality, swap the model

When a frontier model handles one narrow, repeatable job, you're paying for general-purpose capability you don't use. Fine-tuning can move that task to a smaller open model while keeping the quality you already accept, so inference no longer scales at frontier-model token prices.

Training produces a lightweight LoRA adapter on top of a frozen base model (Qwen3, from 4B up to 235B). You get the task quality of fine-tuning without the compute cost of a full retrain, and the adapter serves on any vLLM deployment with a single flag.

How it works

1. Select LoRA fine-tuning.

2. Upload your training examples in standard JSONL. If you have a golden set, add it and the run will benchmark against it.

3. Choose a model size and set a budget. There is no GPU setup and no configuration file to manage.

4. Start the run and follow training progress step by step.

5. Get the adapter, the loss curve, and the evaluation results.

What we saw in production

We use the same approach in one of our own production pipelines. Mindrift, Toloka's contributor network, parses CVs at volume. Mindrift, Toloka's contributor network, parses CVs at volume. The task is narrow and previously ran against a frontier API. We rebuilt the pipeline using synthetic labels from a frontier-model ensemble, a strict human-labeled hold-out set, and a small open-weight model fine-tuned on that data.

• 18× cheaper inference on the same task — from 15$ per 1,000 CVs to roughly $0.80 per 1,000 CVs.

• 0.94 F1 on the human-labeled golden set, up from a 0.85 frontier-model baseline and slightly ahead of the latest frontier model we tested, at 0.93.

• Over 100% price-performance improvement against standard frontier models.

• A few days end to end, from data curation to a deployed endpoint.

• Cost became effectively fixed compute instead of variable per-token spend — predictable as volume grows.

These are results from our own internal production case, not a client project. We are sharing them because the self-serve tool uses the same method.

The full write-up, including the labeling method and the hold-out design, is in Overperforming Frontier Models by 100+%: Building a Production CV Parser with Shopify's Tangle.

Prompt gisting: compress the static part of your prompt

Most of a production prompt is static: the instructions, rules, and few-shot examples. Only the input at the end changes. Without gisting, every request processes the full prefix again.

Gisting trains the model to retain the instructions instead of reading them on every request. It distils a long, static instruction prefix into 8 to 64 learned tokens, depending on the regime you choose. The base model stays frozen, and your existing input/output examples provide the training data. At serving time, the soft tokens are baked into real vocabulary tokens on vLLM's optimised, prefix-cached text path. Each request then consists of a few dozen learned tokens plus the dynamic input.

How it works

1. Select Gisting.

2. Upload your prompt and training examples as standard JSONL — with only the dynamic input in each example. Leave out the static prompt you want compressed; that's exactly what the soft tokens learn to replace.

3. Choose a model size and set a budget. There is no GPU setup and no configuration file to manage.

4. Start the run and follow training progress step by step.

5. Get the compressed prompt, the loss curve, and the evaluation results.

Results from a production workload

We measured gisting on a real production workload with a 5,000-token instruction prefix on a Qwen base model:

• 5.3× fewer prompt tokens per request, from 5,019 down to 939 on average.

• 15% lower GPU cost per sample, and that holds at any $/GPU-hour.

• 18% more throughput in a saturated serving benchmark — and the gain compounds under load, reaching 1.8× throughput at 256 concurrent requests.

The throughput gain matters most under load. With a warm prefix cache, a single idle request sees little effect from a long prompt. The difference appears when hundreds of 5,000-token contexts compete for KV memory and attention bandwidth:

For busy serving environments, the working range is roughly 1.3× to 1.8×, depending on how long your outputs are: decode-heavy workloads sit at the bottom, short-output extraction and classification at the top.

These results come from one production workload and should be read as an example, not a guarantee. Every gisting run repeats the benchmark on your prompt, so you can evaluate your own numbers before moving to production.

What a run costs

You pay for the GPU-seconds a run actually uses — not the estimate, and never for a rejected run.

Based on your parameters, we estimate the run time and place a corresponding hold on your balance. When the run finishes, we charge only for the GPU-seconds used and automatically refund the unused part of the hold. Every run is capped at six hours, and every completed run reports its exact GPU-seconds.

If there is a problem with your data, we reject the run with a clear explanation before any GPU is allocated. You get the explanation in seconds, with no charge.

Training is a one-time cost; the savings continue at inference.

Where this fits

Toloka has already moved evaluation and data labeling for fine-tuning into self-service. Now we're bringing the training step into the platform too, so more teams can run fine-tuning experiments themselves. You can upload a file and start a run without first talking to our team.

Our team is also happy to support you beyond a single training run, with work such as data collection and labeling, evaluation design, or a full experiment ladder from prompt optimization through to reinforcement learning. The self-serve tools give you a way to test a small model on your task first.

Getting started

Use a JSONL file in the standard OpenAI chat fine-tuning format. Each line contains one example and ends with the assistant message the model should learn to produce:

{"messages": [{"role": "user", "content": "<input>"}, {"role": "assistant", "content": "<target>"}]}

For gisting, the user message should contain only the dynamic input. Leave out the static prompt you want compressed; the soft tokens learn to replace it.

Upload your data, pick a model size and a budget, and start a run — fine-tuning or prompt gisting. Results arrive with copy-paste inference instructions.

Learn more about fine-tuning and prompt gisting