New LLMs Are Unlocking Robot-Use Agents
In a Nutshell
New LLMs like OpenAI’s Astra can now control robots by writing code directly, bypassing slow VLA fine-tuning. Their improved spatial reasoning stems from pre-training on computer-usage data (CAD, screen interactions) that teaches 3D concepts transferable to physical control. The path to general-purpose robots in ~2 years hinges on fast “in-context” planning, distilling learned skills into reusable policies, and managing expanding skill libraries.
These notes were generated by AI and may contain inaccuracies.
One of the biggest surprises of the past few years has been the ability of programming agents to generalize across different domains. Researchers in this field are showing that this also includes controlling robots. This prompted MIT professor Philippe Isola to suggest in a recently published article that we may be entering an era of robotic agents, where general-purpose models can make different robots more capable. The founders of Waddle Labs and RoboCurve were invited to discuss making robots more capable using large language models.
Jaime from Waddle Labs, with Vincent, are working on building large language models that control robots through two things: building a system that allows large language models to do this effectively, collecting data, and then using that data to train better models. Jay from RoboCurve stated they are a company specializing in physical artificial intelligence. They measure everything, any robot, any model, including large language models, as well as traditional approaches such as vision, language, and motion models, and world motion models. They evaluate all types of environments such as hands, grippers, arms, humanoid robots, quadrupeds, and all kinds of objects.
Videos of both companies have been widely shared on Twitter and are mentioned in Philippe Isola's article about agents using robots. The videos showed things like the ability of large language models to unpack lids, communicate between multiple robots, and perform tasks such as opening a pen.
The RT2 research paper established models of vision, language, and movement. One of the first successful approaches to using artificial intelligence in robotics is the RT2 research paper, in which they used a linguistic model pre-trained on web texts and images, and used it to control robots. It's a finely modified version of a language model, and instead of outputting English, it outputs what is called the influential terminal mode, which are coordinates that can then be translated into articulated commands that can control robots. In some cases, it is very similar to what we are seeing now with large language models, only instead of fine-tuning, the models are good enough to do it right out of the box.
This is similar to the comparison with a "chain of thought" (COT) moment. When testing GSM8K, the model had to produce four hash marks and then answer, and then the end-of-text symbol (EOS). The model lacked the ability to use the chain of thought. When allowed to say "Okay, let me calculate 8 minus 5 equals 3," and so on, it allowed the model to do this kind of thinking before giving an actual action or answer. Similarly, basic VLA models such as RT2 are essentially models that should output a direct action. There is no option such as "I cannot allocate more computing power to a more complex task." Now with the sequence of code ideas, the model can say: "Well, even if you're using the example where you don't actually output the code, you're just giving the order to Astra and letting it think and think and think and then output a procedure."
The discussion centers on the "bitterness lessons," where if you give the agent, that is, if you give the AI model more autonomy, if you free it up a little and give it more resources, it can actually do many of the things that we fine-tune it to do. What's really interesting about the "bitterness lesson" here is that VLA models, in their design and architecture, are built on top of linguistic models as well. Theoretically, it can deduce, and it can write code. Perhaps the best lesson here is not necessarily in the structure you build, but in which data is most useful. True, people have been trying hard with the Vision, Language, and Actions (VLA) data button for years, and we are seeing very slow progress.
The lesson here is that we have a pattern of data that we know works very well, and that's the idea of moving across different patterns. The best way to benefit from the lesson is to say: Let's choose data in which we know this pattern is blessed and distinctive. There is a lot of data. These linguistic models are well understood, and then we use this as a means to open up many other areas as well. The reason RT2 is such a good model is that it uses a linguistic model that takes advantage of the pattern of all the data that the linguistic model has been trained on. Therefore, all those images and tags from the web actually improve the VLA model compared to training a basic robotics model without any prior training. In the case of RT2, which is this early VLA approach, you benefit from pre-training.
The limitation in the fine-tuning approach to taking action that can now be bypassed if you can write the code directly is the discrepancy in the data. The models have become much better at using tools since the advent of RT2. Now these models can write code much better. Therefore, they can now write complex policies in code form. They are also much better at using tools to explore the types of environments they are in, such as the arms they can reach. Much of this comes from learning within context. The main difference between these models (VLA) and GPT-6 or Large Language Models (LLM) is not necessarily the structure, but rather the approach taken in training. The goal is to learn from the best lesson and take advantage of all types of data, inject computer usage data into robotics models, and inject programming data into robot models.
Looking at "code as policy" was a good indication that we were moving in the right direction. The "code as policy" research was published when programming agents were just starting to work, still in the era when people would put comments and automatically complete blocks of Python code. "Voyager" was one of the first programming agents, requiring good use of tools and creation of tools during operation. This is called a code. You have built-in Python functions. Those are the tools. You have the order, "if", "for", "while", and all of these things, and you need to use those tools to compile a new tool called, for example, "francois.py." Now that's a new tool and you can use it. Voyager was most famous for doing this for the game Minecraft. They created tools that they could call during gameplay to help them play the game better and compress the thinking and experience into a new tool that they could call later.
The research papers on code policies, especially those coming from the "Google DeepMind" team, were amazing because what they did was create these kinds of functions like picking up an object, lifting it, or moving into a certain position. They provided a list of these functions to the programming client in the form of programming functions in the Python language literally. The programming client was then able to write code that used these functions to control the robot to perform very complex tasks. What was most surprising was that the programming clients could do it on the first try. They did not need additional robots data to work with this code because they were already trained on a huge amount of programming data. They already have a sense of what needs to be done first, and what needs to be done second, in order to move a mass inside a container, for example. This ability to execute on a single attempt, and this ability to explore contextually, is what has really motivated much of the subsequent work, including ours, to continue exploring how to apply large language models in robotics.
There is a framework regarding the various ways in which learning can happen, including weighted learning versus cross-context learning. The most efficient way from a machine intelligence perspective for each sample to input learning, say in this context, the result of an action for a situation or the reward for an action for a situation, to bring it back into the policy. There is learning in context (ICL). This is learning in the context that we are describing here. Learning is where you simply attach it. This is how most people use Large Language Models (LLMs). They just say, "Oh no, don't do it that way. Do it this way." And then it remains in context, then you can remember and you can repeat. This only really works if you have trained the large language model, or at least subsequently trained it to be able to learn and improve, and that self-reflection, which resembles old literature, actually allowed for multiple roles in the training group. If you don't have that, he doesn't learn, and he doesn't learn in context.
Even then, there is a limit to how good In-Context Learning (ICL) can be. When taking a large language model that has been trained on, and excluding a task, say GSM8K for simplicity, then doing In-Context Learning (ICL) for him, and measuring on a validation set how much he has improved for each sample, it is improving significantly and at a very low cost. It doesn't cost much. You don't have to do any stepwise derivative descent (SGD), so you don't have any calculations (flops). Thus, you can adapt and improve very quickly on the verification set. The first is non-routine improvement. It gets worse, then it gets better, then it gets worse, then it gets better quite strongly. The second is that it reaches its maximum limit very quickly. So after about 20, 30 or maybe 40 examples, it basically reaches a state of saturation, and adding more examples to the context does not lead to any improvement. So you are limited by the model's ability to use all of its context intelligently.
From the post-training phase, how many multiple roles has he actually had to be able to improve and engage in self-reflection? Most importantly of all, it is definitely not possible for it to do that once it reaches the context window and the length of the model's context that it was trained on. If it is trained on 100,000, that effectively means you have a context window of approximately 50,000. Once you exceed 50,000, the results will no longer improve. In fact, things are getting worse because the model cannot pay attention to everything. If you could use RAG technology or load information into active memory, this is similar to "continuous primary agent" thinking where you can pull information on the go and similar examples, it's like having a test and having the textbook. You will perform better if you have the reference book available to you to search through. So, this is sure to work, and it also expands much better. Then the last two models will be "LoRA" of order one, two, or whatever. "Laura" at level 10 or 100, up to full training (Full SFT/RL).
The idea of difficulty as a form of specialization within the scope is considered. For example, when deploying a robot in a new environment, perhaps in a humid laboratory where it needs to do a lot of tasks picking up test tubes, you can learn a lot of this in context, but one way to solidify that context for future agents is to package each skill you learn into specific programs. Writing down these skills and recording these memories is a form of unification. It is similar to a kind of distillation of past experiences for the benefit of your future agents. There appears to be a connection to the broader meta-learning literature throughout the history of machine learning. You have a larger model that programs a smaller model to perform certain tasks. Interestingly, these smaller models will actually learn in context. They are like models encased within a framework that perform the task. You can make it smaller, you can make it faster, as long as the larger model is able to do this scale customization for your framework well.
Astra is shown controlling the arms to pick up a cube from the table and place it in the bowl. Astra uses only camera inputs, presumably to be aware of the robot it is controlling, and will write code that controls the robot's individual joints. It uses all cameras and has access to all camera feeds. Instead of code, it's more like calling a tool. Therefore, it sends commands to the robot to control it and place the cube in the container. In this case we are using Astra directly, but if we were to use, for example, the Waddle framework, the difference between using Astra directly for control versus doing so via the Waddle API is that sometimes, having Astra guide the robot directly into the desired position is not the optimal tool to use. If the task is repetitive, you don't want "Astra" in the workshop. You might want to write code that can be run repeatedly and very quickly. Or if it is a task you have performed something similar to before, you should be able to call upon a pre-programmed skill and use it to accomplish the task faster and handle exceptional situations better.
The response time appears to be slow because we are limited by the "Astra" response time. But if you look at the trends, the response time of these models is improving very rapidly. One of the things we've seen is that large language models of the "Viable" class are improving their speed by almost double every month. It's very, very quick. If these trends continue, we may have instantaneous control by the end of the year.
There is a software agent within the loop that runs this. What just happened was that, in a series of turns, Astra received images from the cameras and put the robot's limb into position to move towards them. He does this through several roles to complete the task. This is less about software policies and more about tool calls. The part that appears on the screen is very repetitive. So, you can actually automate a lot of that using code. In addition to actually approaching the bottle and picking it up, many of these things become absolutely inevitable once you have done the task several times. The interesting part is where you build variation within the software policy drawing. We have a few points of divergence. This can happen, for example, when an object is discovered. You might use a visual language model as a tool call-out. Or when something fails, for example, how do you verify that it has failed? How then do you determine the likelihood of doing something else based on the type of failure? For such more flexible responses, we tend to place a visual language model inside a loop to ensure that while the programming drawing seems deterministic, there are points of variation that allow for generalization.
The biggest insight that changed the worldview about how to conduct machine learning and how to cut the rest of the way towards artificial general intelligence came from many conversations with François Chollet. In 2020, and perhaps even as late as 2018 when he talked about measuring intelligence, he discussed a lot the concept of connectivity versus program extrapolation. The transformation simply means that you are learning the "theta" function, which maps values from x to y. Why is that considered a mistake? It is simply slow and inefficient in terms of information. To move from x to y, you need a lot of x and y pairs. If you have too few, you need a lot of inductive bias, and what you really need is a good generator for assigning x to y. So now your theta takes three pairs of n from x and y and outputs the function that maps from x to y. This is the essence of programming code. If you gave a programming interview on a whiteboard, and said, "Okay, this is a programming problem, and these are some examples. Okay, great, write the function." Then we issue a function "F" which will set the values from x to y.
A major reason why those methods don't work well is that inductive bias isn't right. You are transferring the difficulty of the problem from finding that assignment to finding the correct inductive biases to assign it to this smaller code to do the assignment later. But finding that set of inductive biases is extremely difficult. So, perhaps this is what "Astra" offers. Even in cognitive science, there is a huge shift in the literature from using specific symbolic neural methods to simply using "Astra" to write code to do the mapping. This is the symbolic nerve. We have neurons that then produce symbols, which are programming codes.
This is a very good transition to the last topic, which is about drawing inspiration from the research paper by Filippo Occola, which he titled "The Platonic Representation Hypothesis." It is "Platonic" because it is a reference to "Platonic cave" and the idea that you see shadows of a diverse reality. The point he's making here is that there is extensive evidence showing that different language models and data representations actually learn these spatial assignments between similar things under different and overlapping training policies. Thus, the idea would be that the more these systems are trained on larger amounts of data, the more they converge towards a kind of consistent mapping of the world. Perhaps this means that we expect language models to get better and better at things over time. This will make it useful for new tasks such as controlling robots. This representational hypothesis is actually essential to the worldview or to the vision of the companies.
Very strong language models will have representations of the world that are very similar to strong robot models. If you have a very strong language model, you also have a very strong robotics model. Using Platonic representation, the best lesson is that the best embodiment is the best lesson, because you only need one very strong model regardless of the architecture, and it will outperform any slightly weaker specific models.
Newer models like Astra seem to be much better at spatial intelligence, controlling Blender programs and creating 3D images. There is significant improvement in certain tasks according to the standards. The question is what changed in OpenAI's approach in pre-training and post-training phases that made these models significantly better compared to models that were only six months prior.
The basic lesson is still benefiting programming and mathematical problem-solving, but spatial intelligence has not yet been reached. Astra excels at vision capabilities and has probably been pre-trained on much more computer usage data than ever before. It has been pre-trained on a lot of computer-aided design (CAD) data. All these types of data seem to teach the model a similar understanding of the physical world as much robotics data might.
Computer usage data is not entirely intuitive because it tends to be just a click on the screen. Training a model to use only a computer will not make it applicable in the field of robotics. However, feeding a large model like Astra with computer usage data, such as dragging a cursor on the screen to rotate an object in the Blender design program, gives understanding of how to think about spaces. It tells about concepts like up and down, left and right, and all those concepts needed to control a robot. This is why this data helps so much in making these large language models much better at being used by robots.
In the robotics community, people are experimenting with more different types of data, such as personal perspective videos. The field has moved from simply operating remotely to a broader set of this data, because it is not just robot data that can teach the model how to use it. The extreme approach would be feeding in every type of data: programming, computer use, and personal perspective, all in the same model. This is the way to reach the most capable robotics agent.
Going back to the 1980s at Xerox PARC, the graphical user interface was made more like the physical world so that interaction could occur. This built an environment that was actually helpful for robots to learn how to interact with the physical world. File systems, files, folders, and graphical interfaces for programs like SolidWorks and Autodesk were created to rotate things and make their simulations similar to the physical world. Now it operates in the physical world.
There was research from Princeton University and Cloud Robotics dealing with designing a suitable binding system so that the tools that interact with the robot look like computer-aided tools. An agent pulling a pointer to control the robot's destination improves the ability of the large language model to perform these physical tasks.
Given continued improvement of underlying models, capabilities that are difficult now will become increasingly possible or even trivial after a few months. There is consensus within frontier labs and robotics companies that general-purpose robots will exist within the next two years or even sooner. When general-purpose robots are mentioned, it means something that can perform any instruction in natural language, something a competent young person can do with their hands. It is similar to a ChatGPT moment, but in the field of robotics, where it can generalize to invisible tasks and environments.
For the Wild Labs team, there are many obvious challenges to address in about two years. Response time is a big problem. If Astra thinks through every single step in the episode, that is really slow and not economically beneficial. The question is how to incorporate an initial attempt at contextual learning from Astra into a faster skill or policy that can run repeatedly with very high productivity.
Relationships with humans will become more like where the optimal course of action may be to quickly put a new situation, procedure, and reward into context, but then there should be some sort of period of respite. Almost everything intelligent goes to sleep. During sleep, the pressure occurs. The strange thing that happens out of the hippocampus is short wave-like ripples to both lobes, and there are strange pathways of memories that have been compressed throughout the day for weight training.
The right course of action might be similar to Dagger, a data-gathering framework in classic reinforcement learning, where a set of data is collected and sorted, then used to update the weights file, and then there is an association from those experiences to an updated weights file, perhaps not Astra, but perhaps your own models.
It looks exactly like Dream Coder, which had a library of skills, and during the sleep phase would restructure everything into a more compact representation. Similar things are happening now, not with the same rigor as before in the software field, but restructuring pathways and restructuring skills. Many research papers are about robotic code as a policy, with a growing library of skills, and it is really hard to know how to trim them and organize them.
Going forward, managing a growing content of skills and publishing data will be very interesting.
Keep Y Combinator in your library
Save the videos and channels worth coming back to, and find them again in one place.
