Modern LLMs as robot control policies

Intro

Controlling a robot typically means writing specialized code for each task (pre-Transformer approach), or collecting data and training VLA or WAM or any other transformer-based model.

Using a bimanual Trossen AI robot, we tested and compared different robot control approaches using GPT-6 Astra and Jev in fixed experimental setups.

Key takeaways

• GPT-6 Astra’s visuomotor approach achieved 90% success on the harder object-selection test when compared with 30% for the classical control stack.

• Selecting predefined motion primitives from text descriptions proved less effective. Astra achieved 25% success on the simpler test; Jev achieved 0%.

• Running Astra as a visuomotor policy required adjustments for latency, with each episode costing approximately $5.

• The results suggest a possible direction for further experiments combining LLM-written control code with end-to-end policies.

Where general-purpose models fit in robot control

In the latest wave of robotics (after the Transformer architecture), the general approach to controlling a robot for manipulation tasks is to use a transformer-based end-to-end model, like a VLA (π0.5 [1], GR00T N1.7 [2, 3], RT-1 [4], OpenVLA [5]) or a WAM (DreamZero [6]).

Achieving reasonable performance on your hardware and tasks will likely require collecting data on your setup and fine-tuning the model. You may even need to pretrain from scratch, depending on how much your setup differs from the lab environments used to train the open-source models.

In September 2026, a new generation of agentic models appeared, including GPT-6 Astra, Anthropic Fable 5.1 and Opus 5.5. We also included Typesafe Jev, a classifier model, in our experiments.

This new generation led to a natural question: can we use them to control a robot instead of going through the expensive process of collecting data and training a VLA?

And if we can, the next natural question is how? What role should a general LLM play here? Should it only write deterministic code for us? Or should it replace the VLA entirely and work almost end-to-end?

There’s a strong article and discussion on this here [7] and also here [8]

So, for this article, we ran a few experiments with different approaches to using a general LLM (GPT-6 Astra) as a robot control policy.

GPT-6 Astra classical stack aka model-based aka good old engineering

This approach is driven by the ability of general LLMs to write code. Before transformer-based models, programmers would manually write code to control robots, building pipelines for object detection, trajectory planning, and trajectory execution. In other words, they would do good old engineering. Now, with this approach, we simply tell our coding agent to “build this pipeline for us”.

And we can now do this much faster than before (a couple of evenings for one IC now, versus several months for several ICs before).

Here’s an example of how GPT-6 Astra built such a pipeline for our Trossen AI robot.

We detect objects using SAM 3, build a 3D scene, and plan trajectories.

Across all experiments, we keep the existing robot controller fixed. The controller lets us command Cartesian tool center point (TCP) positions or velocities and read TCP positions alongside RGB images and depth data from the cameras. We change the pipeline that turns those inputs into movement commands.

How we measured model performance

We measured model performance on two tests in our lab, using the bimanual Trossen AI robot.

Each test has 20 fixed scenes.

In the first test, there is always one object and one container in the scene. The robot must pick up the object and put it in the container.

In the second test, there are three objects and two containers in the scene. The robot must pick up an object by name and put it in the correct container. So that the second test is a harder version of the first, we add conditions for which object to pick and which container to put it in.

For the first model, results were:

Here you can see the episodes from both tests and what the scenes look like:

What’s notable is that it’s good at identifying the right object: an approach with a separate segmenter like SAM 3 works much better than trying to do this by collecting data and training a VLA. In our experience, it’s much harder to teach a VLA to identify an object by its name in the prompt.

Most errors come from the fact that we still have to handle every case manually, and it’s hard to anticipate them all. Therefore, it fails almost every time on objects with less standard shapes (clamps).

So, as we see it, the main problem with this approach —even when LLMs write the code — is that you have to account for all the corner cases in advance, which makes it hard to generalize.

Here’s how it works.

Examples of successful episodes.

Examples of failed episodes.

GPT-6 Astra visuomotor policy

Now, what if we replace our end-to-end model, like a VLA, with a general LLM, GPT-6 Astra?

In this experiment, we do exactly this.

One of the bottlenecks is model latency, as we have to input our RGB images and depth, and output a trajectory (multiple tokens). Even with ultrafast mode on GPT-6 Astra, inference takes > 1 sec. 

We had to find a way to use our 25 Hz control stack without predicting the entire 25 Hz trajectory to reduce the number of output tokens. So we predicted a 5 Hz trajectory and then interpolated it to 25 Hz. We also compressed the depth and downsampled the images to reduce the input size. The overall architecture looks like this.

When we started the experiment, we didn’t expect the results we got. Our expectation was that something closer to a model-based approach would perform better. But surprisingly, GPT-6 Astra has strong spatial reasoning and can perform well on our tests.

The visuomotor policy achieved the following success rates:

The difference between 60% and 65% on Test 1 falls within run-to-run variation. The clearer separation came on Test 2, where the visuomotor policy achieved 90% success compared with 30% for the classical control stack.

See what the actual rollouts looked like here:

Some successful episodes:

Some unsuccessful episodes:

From what we can see in our first tests, this model’s ability to generalize is significant. Not to mention we get almost everything out of the box. Our harness here is mostly for handling its latency and low-level control.

The main disadvantages are its latency and cost. One episode cost ~$5.

GPT-6 Astra / Jev primitive selector

We compared GPT-6 Astra with Jev by asking both models to choose the robot’s next movement. Each choice came from a set of predefined motions, known as motion primitives.

We classify our predesigned motion primitives. A motion primitive is a small, simple motion that the robot can do. For example, move 10 mm in +y or rotate 30 degrees around z. Some of the primitives were selected from 20 example episodes. In total, we had a catalog of 54 motion primitives.

This approach is a simplified version of ASPIRE [9]. In ASPIRE, a coding agent writes robot programs and builds up its own library of skills. In our version, the model only picks the next primitive from a fixed catalog.

As Jev doesn’t support input images, we had to encode our RGB + depth input in a structured text format to make the comparison fair.

We use SAM 3 to describe the scene, create a structured scene description, and pass it to our classifier model, which selects the best motion primitive. All that’s left to do is execute the primitive.

You can see the overall architecture in the following diagram.

GPT-6 Astra managed to do some tasks successfully, while Jev didn’t. Most of the time, Jev selected the wrong primitive and got stuck in a loop.

We attribute the poor performance to the fact that we have to compress our input into a text description, losing   information. And Jev performs worse than Astra at interpreting this compressed information.

Latency and success rates

Ss we have to run inference on SAM 3 and then on the LLMs, the latency of the whole system became even higher for GPT-6 Astra than in the previous approach. We used ultrafast mode for Astra here too. Jev’s latency was significantly lower than Astra’s.

Metrics:

We didn’t run Test 2, as it wasn’t worth it due to the models not being able to reach any reasonable SR even on the simpler test.  We decided to save our tokens.

You can check the tests here:

Some successful Astra episodes:

Some unsuccessful Astra episodes:

Some unsuccessful Jev episodes:

Summary Metrics

Conclusion

Based on these experiments, the approach that showed the most potential is using LLMs in a way similar to VLAs, more like end-to-end: the model gets the robot and environment state and outputs actions.

Perhaps the optimal approach will be a combination of deterministic blocks and end-to-end models. And we can use the models themselves to find that combination.

Looking at our experiments, that would mean combining the first and second approaches.

One way to frame the progression is as follows.

• Before end-to-end approaches, we optimized in the space of control code to build robot control models (humans did this manually), but this was expensive in terms of engineering time.

• Then, after the Transformer architecture became popular, optimization shifted towards model weights (which requires a lot of data).

• With LLMs that write code well, optimizing code has become cheaper (as in our first experiment), since we can now optimize it automatically. For example, in a simulator, or even on a real robot, like in our first experiment.

• At the same time, LLMs themselves can, to some extent, run inference like VLAs (see our second experiment).

• We can think about optimizing both the code and the weights of the end-to-end parts at the same time (for example, for contact manipulation, where it’s hard to account for every case in code and/or simulation).

Our experiments suggest that general-purpose models can take on part of the work involved in robot control without task-specific training. How much depends on the role they play in the system. The best results came when GPT-6 Astra could use visual input to generate actions, though latency and cost remain obstacles.

References

[1] Physical Intelligence et al., “π0.5: A Vision-Language-Action Model with Open-World Generalization,” arXiv preprint arXiv:2504.16054, 2025. doi:10.48550/arXiv.2504.16054

[2] NVIDIA, “GR00T-N1.7-3B,” model card, Hugging Face, 2026. [Online]. Available: https://huggingface.co/nvidia/GR00T-N1.7-3B

[3] J. Bjorck et al., “GR00T N1: An Open Foundation Model for Generalist Humanoid Robots,” arXiv preprint arXiv:2503.14734, 2025. doi:10.48550/arXiv.2503.14734

[4] A. Brohan et al., “RT-1: Robotics Transformer for Real-World Control at Scale,” arXiv preprint arXiv:2212.06817, 2022. doi:10.48550/arXiv.2212.06817

[5] M. J. Kim et al., “OpenVLA: An Open-Source Vision-Language-Action Model,” arXiv preprint arXiv:2406.09246, 2024. doi:10.48550/arXiv.2406.09246

[6] S. Ye et al., “World Action Models are Zero-shot Policies,” arXiv preprint arXiv:2602.15922, 2026. doi:10.48550/arXiv.2602.15922

[7] @Ken_Goldberg, [Post and discussion], X. [Online]. Available: https://x.com/Ken_Goldberg/status/2100986412762087909. Accessed October 16, 2026.

[8] @GeorgiaChal, [Post and discussion], X. [Online]. Available: https://x.com/GeorgiaChal/status/2101335513558929868. Accessed October 16, 2026

[9] R. Lu et al., “ASPIRE: Agentic /Skills Discovery for Robotics,” arXiv preprint arXiv:2607.00272, 2026. doi:10.48550/arXiv.2607.00272