Fine-tuning for agentic workflows: Building a production CV parser with Shopify’s tangle

Every week, Toloka’s Mindrift network receives thousands of CVs from experts looking to join our contributor community. To match these experts to projects at scale, we need to turn unstructured CVs into a structured, schema-validated JSON payload—extracting specific fields and mapping them to our internal expertise taxonomy.

It’s a complex NLP problem, but a highly bounded one. And that combination—hard, narrow, and strictly defined—is exactly where routing to general-purpose frontier models becomes an architectural bottleneck. In an agentic workflow, using a trillion-parameter model for a deterministic extraction task is latency-heavy, expensive, and ultimately overkill.

Task-specific fine-tuning is the countermeasure. By training a smaller model exclusively for this node, we matched frontier-level performance at a fraction of the cost. Here is how we built our production parser, and why this pattern is critical for scaling enterprise AI systems.

The pipeline architecture

Our previous production parser, which relied on an earlier-generation frontier model, was hovering around a 0.85 on our human-labeled golden set (per-field multiset F1). While functional, it left a significant quality gap compared to what the latest state-of-the-art models can achieve today. However, at our volume, simply migrating our pipeline to the newest, most expensive frontier APIs wasn't viable due to unit economics. Closing that quality gap without blowing up our cloud bill was the target.

To orchestrate the fine-tuning data engine, we used Shopify’s Tangle, an open-source tool built for automating pipelines with complex business logic. Tangle proved to be a natural fit for structuring the ML lifecycle: orchestrating data ingestion, executing the fine-tuning job, and running automated evaluations.

Our methodology leaned heavily on distillation and rigorous evaluation:

• Synthetic Label Generation: We built our training set by routing thousands of CVs through a mix of frontier models combined with automated validation checks. This generated cleaner, more consistent training targets than any single API call could produce.

• Strict Hold-Out Evaluation: A separate, human-labeled golden set was held out entirely from the training loop to ensure an unbiased evaluation of the fine-tuned model.

• Deterministic Output: We then applied standard fine-tuning techniques to a small open-weight model, with additional safeguards at inference time to keep output strictly schema-valid.

The end-to-end pipeline, from initial data curation to deployed endpoint, took just a few days.

The ROI of specialization

The resulting fine-tuned model achieved a 0.94 F1 score on our golden set, capturing 96% to 98% of frontier API quality. This wasn't just a cost optimization; it was a massive jump in production accuracy from our previous 0.85 baseline.

The unit economics of replacing the frontier API with a self-hosted, fine-tuned SLM completely changed the cost structure of this workflow:

The remaining fractional gap to parity with frontier models is actively being closed through targeted human-in-the-loop (HITL) labeling and continuous refinement of our evaluation rubrics.

Why this matters for multi-agent systems

If you are building multi-agent systems or complex AI pipelines, your workflow is likely full of tasks just like CV parsing: repeatable, high-volume, and strictly scoped.

Relying on frontier models for every node in a workflow is a prototyping strategy, not a production strategy. Toloka's enterprise fine-tuning thesis is built on this reality. By combining our data engines and ML expertise, we help enterprises fine-tune generic, bloated API calls into fast, specialized models shaped entirely around your specific business logic.

What’s next

CV parsing is just one of several high-ROI fine-tuning use cases we run at scale. If your engineering team is managing an agentic workflow where a frontier model is eating your margins or slowing down your pipeline, let us help you build the data engine to fine-tune a model that lowers costs, reduces latency, and drives accuracy.