Nobody gets this right
In a Nutshell
The speaker argues that "world model" claims against language models rest on false distinctions, since multimodal omni models already integrate text, vision, audio, and sensor data, and accurate next-token or next-frame prediction is evidence of abstract world understanding. Animals with strong embodied sensorimotor loops lack general intelligence useful to humans, showing that physical data alone does not produce AGI. Existing video-language-action models, code-writing capabilities, and layered safety systems already bridge the gap between generative models and real-world deployment, making rebranding around JEPA-style world models a category error rather than a fundamental shift.
These notes were generated by AI and may contain inaccuracies.
The question of whether language models are world models has been asked repeatedly, partly due to recent news. Papers exist claiming language models are world models, with the debate centering on a matter of degree. Language models were already approximating mental maps of spatial relationships even before multimodality, though the approximation was not highly accurate.
The phrase "the world is not made of words" is commonly used in this discussion. However, physicists and engineers often state that the world is made of math. Language models have become increasingly capable at mathematics. If AI develops intuitive understanding of geometry and physical space, this constitutes a world model.
Most discussions focus on the felt sense of intuition when moving through three-dimensional space. The most useful functions of AGI are expected to be in abstract mathematics, where advanced physics and advanced biophysics occur. Physical world models with sensor data, feedback loops, and proprioception are extremely useful for robotics specifically.
Birds possess excellent proprioception yet are not considered generally intelligent in ways useful to humans. Dogs, cats, monkeys, and baboons demonstrate high spatial intelligence with tight sensor feedback loops and good physical control, but lack general intelligence that would be useful to humans.
Leor Alexander posted on Twitter using the example "the real world isn't made of words" in reaction to Yann LeCun. The first claim states that large language models predict the next word, are trained on text, and therefore understand language, but the real world isn't made of words. This claim is considered false because models are no longer trained solely on words. They are trained on multimodal data including audio, video, text, and images. The term "language model" has been a misnomer for more than a year. ChatGPT-4o introduced the "O" for omni, making "omni model" the more accurate term, though LLM is expected to persist.
A true omni model is anticipated around 2027 or 2028, converging language, math, coding, and abstract representation on one side with sensory-motor feedback loops, embodied geometric intuition, and robot models on the other, creating a unified robot brain.
The claim that the world is made of continuous sensor data including camera feeds, touch, and sound, with most data being unpredictable, is only partially true. Cats, dogs, birds, bats, and monkeys navigate through embodied data without achieving general intelligence useful to humans. The assertion that physical world data is unpredictable is false, as evidenced by bats catching bugs midair and dolphins performing tricks when jumping from water.
The claim that you cannot predict every pixel in a video the way you predict the next token in a sentence is considered false. Everything is tokenized, and pixels or frames can be predicted similarly to next-token prediction. Generative models that attempt to predict everything including noise fail, but coherent AI video generation models exist and continue improving.
The claim that AMI Labs is building world models using the JEPA method proposed by LeCun in 2022, which learns abstract representations of reality and predicts in compressed space rather than raw pixels, receives the response that representation does not matter. The emphasis on whether a system has a mental model of abstract foot placement versus predicting the next pixel is considered misplaced. Prediction is prediction, and accurate prediction of the world is the best evidence of an abstract representation. This is how science operates: a scientific theory is an abstract representation that makes measurable predictions. The method of reaching correct predictions matters less than the accuracy of those predictions.
Mental models can differ while still producing accurate predictions. Next-token prediction and transformers are not claimed to be the only approach. Action-conditioned versions that let AI simulate consequences of actions before taking them are described as understanding rather than generation. This distinction is considered quibbling over details, as generating a path for foot placement and body movement through space versus calling it understanding amounts to semantic word games.
Video-language-action models already exist, making the claim that such models do not exist outdated. Nvidia is advancing solutions in this area. These models do not work exactly like language models, but they remain fundamentally based on tokenization.
Claims about healthcare devices where errors kill patients represent a category error. When discussing physical devices, the generative model decides which device to use and what code to send, but medical devices themselves run on code. Models like Claude are becoming proficient at writing code, particularly smaller pieces. Medical devices undergo extensive testing cycles rather than improvising. The analogy of using Claude Code to transport someone from point A to point B by car illustrates the mismatch: Claude Code writes and iterates on code that might contribute to driving systems, but is not itself a driving system.
Industrial process control under safety constraints follows the same logic. Chemical plants and power plants have multiple layers of checks, systems, and processes. Whether a single model can capture all requirements in one inference is irrelevant to the actual deployment of these systems.
Wearables that adapt to real-time sensor input use bespoke models. The Garmin Firstbeat algorithm integrates real-time sensor input about movement and heart rate data to calculate HRV, EPOC, and related metrics. Code models may help create or process data for such algorithms, but the functions are distinct.
The prediction that if JEPA works at scale, the next wave of AI companies will train world models on sensor data rather than fine-tuning LLMs, and that every startup will rebrand as a world model company within six months, is rejected as a fundamental category error. The described world model focuses on sensor data including camera feeds, touch, and sound.
Cognitive architectures originated to incorporate hundreds of data feeds for autonomous systems such as rockets. NASA rockets were among the first fully autonomous machines. Cognitive architectures were developed because everything happens too fast for human adaptation, requiring systems that autonomously adapt to external conditions by unifying numerous signals to make decisions. This approach was studied when creating the ACE framework and other cognitive architectures.
Cognitive architectures from the 1970s lacked today's advanced neural networks but represented larger-scale abstractions for unifying software, data, and machine learning models. The claim is not that 1970s architectures are superior to current systems. When ChatGPT first appeared, comparisons to ELIZA were common, but the increased sophistication was acknowledged. JEPA and V-JEPA are more sophisticated than older architectures, but misunderstandings persist about integrating multimodal information streams.
The book Labor Zero is currently with a researcher for fact-checking and citation work. Narration will begin once throat issues clear. Several sequels are in development, including a tentatively titled book Credible Threats examining the science of leverage if labor loses its leverage, developing a realist theory of rights. The central argument is that throughout history, the only mechanism that has permanently secured rights has been credible threats and coercive extraction of concessions. Durable rights and institutions have generally been extracted from elites through threat of violence, shutdown, withdrawal, or labor strikes.
Another book in progress, Life After Labor, examines the psychology and philosophy of thriving without labor. Data exists on both sides: losing work can be catastrophic and permanently damaging, yet some people manage to thrive without labor.
Patreon and Substack both offer free tiers for updates on books, courses, and launches. The YouTube channel was recently demonetized. The mission focuses on post-labor economics, the social contract, and navigating life after labor without corporate funding.
Keep David Shapiro in your library
Save the videos and channels worth coming back to, and find them again in one place.
