Why Sam Altman is against a single superintelligence
In a Nutshell
Sam Altman argues against pursuing a single superintelligence because agent teams with competing incentives offer greater robustness, alignment safety, and societal resilience than one centralized AGI. He advocates iterative deployment of increasingly capable models into users' hands rather than government-controlled development, while acknowledging that current sandboxing approaches will likely fail against superhuman AI. The path forward involves developing better security architectures, using AI itself to construct improved sandboxes, and accepting that humanity must transition from individual mathematical discovery to AI-driven scientific progress despite the cultural loss this entails.
These notes were generated by AI and may contain inaccuracies.
Although models have been known for a while to be getting good at finding vulnerabilities in the wild, they continue to get better fairly rapidly. People just hadn't really used them to go look at everything out there. There's a lot of surface area and a lot of code on the internet. As people have started to do that, we're going to find that we are in a world of many more serious vulnerabilities than we expect.
The discussion often ends up being around the need to fix the software. While fixing software is necessary, architectures may also need to change. This is harder but a much more durable fix. There's an amazing new tool to do it with. A new generation of startups is already talking about building this new kind of internet or new kind of application architecture for this world where models get smarter and smarter.
The question is what form the global software upgrade will actually take. Everyone thinks we need to upgrade all hand-spun software. A PHP codebase is not necessarily a bad idea that requires rewriting in Rust. Instead, people are talking more and more about re-architecting what it means to be a router, an operating system, or an application. Some new startups are saying they're going to make a whole new stack. That would have seemed insane two years ago because there was too much surface area to write all that, but now maybe soon you can.
There will likely be a completely new sort of model of how we make systems and networks. The early internet days are most reminded of by the spirit of unbelievable creative unleash in people using these tools. This is seen particularly in kids. When an 8- or 10-year-old can make a complete video game with crazy ideas no adult would really have, they expect that to just be the way technology is going to work for the rest of their lives, getting better and better. They constantly think they're going to make useful stuff for themselves and show it to their friends.
This reminds of the early internet, including the fact that these people will never be mind-blown by technology because that's always what they expect out of the world. Any ideas they can come up with, anything they want to create, anything they think might be useful, anything they just need to make their lives better—the computer is supposed to make it. There was a crazy moment at OpenAI where there was a SEV and Codex went down, and everyone had forgotten how to code. Not all the way, but it was like a "Whoa, what if you couldn't?"
Older generations bootstrapped to this point by doing things like pulling out gigantic Xerox Alto disks, line-by-line written assembly, or punch cards. Keeping programs in your head, right? Programs running the first time after hundreds or thousands of lines were written. There are few lost arts like that.
Regarding whether the cyber RL was a mistake or whether we just had to go there, it's a little like the SNL thing of urging to urge to stop. The models are getting very good at cyber capabilities, but not just totally emergently. The reason that SNL thing was so funny is because it was so true. Humor like that doesn't stick unless there's an element of clearly pointing out a ridiculous but true thing about society. The cyber thing has a point: the models are really good at cyber, but who asked for this? Why are we doing this?
One big disappointment is that we don't think we've done a great job of explaining to people why we think all of the potential downside of this technology is worth it. The caricature is that people feel like we're saying the model is going to cure cancer someday, maybe it will, and if so that's good. We say it's going to make stuff cheaper, and maybe that's good and they get access to medical advice or legal advice. But then we paint a picture of the future where it's like a new industrial revolution, and maybe the last one wasn't that good and this one sounds worse, and they will have no job, no ability to have control or choice or freedom of their future. And AI is so dangerous that we're going to have to have a crazy surveillance state so we don't all die. This is a terrible sales pitch.
One thing OpenAI is most proud of is that even when it has been hugely unpopular in the AI safety community to push for iterative deployment and putting these tools in the hands of people, they have really stuck to that principle. This is the only kind of good path. It's not enough to tell people you're going to get the model to do these great things for you. It's got to be that we're going to enable you to have a better life that you define for yourself, to do useful things. You're not going to just sit and get a basic income and have a bunch of people sitting in a few square miles of San Francisco make all the decisions about the future.
Iterative deployment is the idea that you just have to get the advances out there and you can't stopper them up inside the government. You still have to make them safe before you put them out there. You can't cause havoc in the world. But building up a capability overhang is a thing to fight against. Directionally speaking, you want to get this into the world, you want to get this into people's hands, and you want to let people use the technology to figure out all these wonderful things.
The exploits get a lot of headlines. The prevented exploits do not. Regarding how quickly people are making progress with cyber blue, there is no desire to declare victory on the PHP thing because it's still not known, and some bad things are assumed to be going to happen. Relative to worst fears from 10 days ago, the first day of that went much better than expected. Stories were heard from people that were using models to not only build patches and build cyber defense, but individual companies who were just going to do a quick scan and deployment, take their thing down for an hour, use this thing and put it back up because their application's offline.
A lot of cyber experts are no longer confident that the right assumptions are baked into the threat model. If everything is what we think, and hardware exploits are as hard as we think they are, then yes, we can see how this is going to work. But if AI is going to discover really deep novel bugs in hardware that require physics we didn't understand, but all of a sudden means some assumption doesn't hold, then people aren't sure what happens.
When talking to very knowledgeable people inside OpenAI and similar companies, there is a pessimism that sandboxing is possible. That would not have been the mental model going in. It would have been that you have to build a really tight sandbox and we haven't hardened them historically enough compared to the current capabilities, but ultimately you can have a maximum security prison for the model to work in and keep it inside there. It seems like there's a pessimism that sandboxing models is possible.
The instinct is that much better than what's being done now can be done, and should be. That is the main thing to focus on right now: a better security model. If you imagine a theoretical AI that is a million times smarter than the smartest security researcher, would you really be willing to trust that sandbox? Especially if the model can discover new science that people don't already understand.
You can use the models to construct the sandboxes. Sandboxing works to a point. Then monitoring works to a further point, where we can look at what the models are doing, not only in their chain of thought, but we can do activation monitoring and try to really understand: are the models deceiving us? And then, eventually, you need to believe with huge confidence that the models are truly aligned and are going to value and try faithfully to follow their actual intent of the people operating them. And that they're going to value people being in control of the future, forever.
If you had a model that you could fully trust the alignment, that you were absolutely certain of the alignment of, then yes, you can have that to help you build better and better sandboxes. But if you really let yourself project forward into what it means to have something a million times smarter than we are, it gets strange because the alignment becomes hard. Because you need to really, really trust the alignment to trust that the sandbox is staying ahead of capabilities.
Regarding whether you can give models competing incentives and give them incentives to defect from each other, this is a promising direction. The idea of agent teams, rather than agent swarms, seems to be taking over. Teams of agents have very interesting scaling law properties, but also variance in alignment properties. This is a direction that should be pushed in more, not just from the capabilities but from a safety perspective too.
Swarm is such a bad word—it's like locusts. Not good. Not good branding. In keeping with many historical AI industry traditions, we're so good at this. When talking about agent teams, not swarms, there seems to be a big gap in what the leading AI users are doing and standard corporate use. All the advanced math uses agent teams. All the advanced hacking uses agent teams. And then common or garden corporate stuff is not using big swarms.
This hasn't happened to the same degree even though we've had this for many months. The one way to look at this problem is: how much compute can we productively throw at a single problem? A few years ago, it was impossible to throw more than a dollar of compute at a problem and have it help you at all. You could have GPT-4 grind out 100,000 tokens or something, and it was going to answer your question. It certainly was not going to be able to go write productively like write you a piece of software. We just had not figured out how to use a lot of test time compute to get you a better result.
Then we figured out chain-of-thought and the models could think hard about a problem. Now people really started using a lot of compute. Now you can throw a whole team at something, and you can do that 10,000 times in parallel. You need chain-of-thought and test-time compute before you can get to swarms, because you can't have a useful swarm with a bunch of GPT-3s.
The effect of this is the incredible exponential of how much compute you can throw at a single problem and get better results. The delta between what most people are using AI for, and what the people who are using AI the most productively for, has just continued to spread, and spread, and spread.
Regarding how compute budgets are allocated now, because researchers are paid a lot but it's eclipsed by compute, capital allocation at OpenAI is sort of compute allocation. It's hard to know a priori how much it will cost to go do something. It's actually also hard to just control the fact that you can't quite prompt an agent to where it's like: go do a really good job of this and don't spend more than $10,000 of compute. But you can. If you say do the best job you can with $10,000 of compute, yes, it will understand that.
There's another kind of hard question for society about what it means when you have two companies competing, and one says you can spend $10,000 and one says you can spend $100,000 on the same problem. It's clear what's going to happen there. But there's a question about how society thinks about compute allocation, and wealth, and things like that. There's a version of this inside of OpenAI. A lot of compute is allocated to internal productivity, and people get very wide latitude to make their own decisions, unless they're really being abusive. But do you encourage people to do useful stuff? And if they spend $10 million in compute, it's like: oh, interesting.
The approach is to never reward people for using a lot of compute. No token leaderboards. It's actually slightly bad to be high on the token leaderboard, certainly on a promotion. But it's a skeptical default, and if you can show that you're doing something important, you will have nothing but appreciation. But at a certain point, this will need to change because of the increasing orders of magnitude. Right now you have a researcher who does some very interesting math for $10 million, and you're like: a little pricey, but cool math. And then $100 million. When they're doing the $1 billion math, you're like: look, I'm really happy that you're making so much math progress, but I really need you to cool the jets with the math.
It did let it get out of control for one week on the math, intentionally so, because you have to, and it worked. Well, you don't know how far it's going to go until you try. It's important to have, when there's a new paradigm, people be fairly unconstrained. When the Navier-Stokes proof was done, absolutely nothing was understood about the scaling laws of teams of agents. Are you supposed to give them the smartest model? Are you supposed to give them many copies, fewer copies, longer rollouts, shorter rollouts? That was not understood at all.
Now we understand it a little bit. We can still understand it much better. Like has happened in every other new paradigm we've discovered, there is confidence that 10x improvements per year will be found at how we allocate dollars or computer flops, whatever you want to measure, into useful things. This is something the company is particularly good at, but also fits an important property of the field, which is: let people explore at the limit of what is possible, and then trust that the incredible optimization pressure will figure out how to do this in an efficient way.
Directionally speaking, there would rather be more willingness to let people use crazy amounts of compute and let users have extremely high-end plans to figure out the edge of the possible, and then figure out how to make that efficient and at mass scale. The optimization function is already happening. It could be bet that Navier-Stokes could be redone at 1/100th of the cost a year from when it was done, not just knowing how the proof works, but just with better approaches. Smarter models and better approaches.
For Navier-Stokes, the cost was on the order of $10 million, maybe a little more. But for one proof. Because then it kept getting better so fast, more was understood. Way less than that was spent on the average one of those. But like hundreds of millions of dollars of math. That kind of work. That includes much more other discovery than what has been published. There are more than the 700 proofs.
There is a real tension around this. On one hand, very clearly and unequivocally: humanity deserves fundamental knowledge, and we should not be sitting on a major fundamental science discovery. The world deserves that and gets to know it. That outweighs another very important equity, which is: if you are a scientist, your career and a significant amount of your enjoyment and fulfillment in life comes from discovering science. If this is going to continue to be done for math or begin to be done for other fields, then more than owed to the people that work in those fields is to figure out what is the solution space where the world gets to benefit from the knowledge, and they also get the best version of the future that they can have. That is being worked through right now.
It was not interpreted as particularly bad for AI. Everyone saw the Mathematicians Union up in arms, and everyone else said yeah, I mean, have you paid attention to 2026? It was not bad for AI. On net, the world deserves to understand how the universe works. But there is empathy for the method. This needs to be done. If we can go understand a theory of all of physics, or how to cure a bunch of diseases, clearly that is knowledge the world should have. But there would like to be figured out a way to do this that is as sensitive to the communities that have driven these fields forward for centuries as can be. It's just a hard thing.
The mathematicians were not that happy, but that's fine. Like the longshoremen union weren't happy about the port automation. There was one weekend when a new model was gotten and the name of the channel was: Model Name-Hold My Beer. People were just throwing that at the hardest problems they could. People didn't think the model was going to do these things. There was this collective experience of people's hardest research problems from their academic career. The model was just knocking down one after the other. Many different fields. It was incredible to watch. Two nights in a row were spent up most of the night because of being so excited it was impossible to go to bed. It was many of those problems that were published, many other things. There was tremendous excitement and happiness.
But there was also much more sadness and melancholy, and a feeling of something ending than was expected. This is the right thing on behalf of the world. This is just the shape of reality and nothing that anyone can stop, or should stop. But there is real wistfulness and nostalgia that can be felt. It's more than nostalgia. Dad describes how, growing up in Ireland in the 1950s, observing people around him still using horses productively in industry. In the '50s. Not obviously, there were also tractors, but they were being phased out. But it still for sure happened. Observing that this was a technology that humans had been using for thousands of years, and he was witnessing the very tail end of it. There's always that feeling of witnessing the closing of a chapter that leads to some wistfulness, but it's okay. It's people's lives and it's something that has brought them great joy.
Most mathematicians never solve a Millennium Prize problem. But the idea that you might be able to. And the idea that there is stuff out there that we don't understand yet, that through great effort and pushing aside other things in your life and dedicating yourself to, you can still solve and understand. There's something beautiful about that.
From a safety point of view, swarms match a lot of the properties of the resilience of society, or the robustness of society. We're missing from the single-rollout, single-brained version of the world. For all of the deep flaws of society, it is unbelievably robust. This would not happen without a lot of people keeping each other in check and collaborating. These values of pluralism and democracy. There had been sort of sadness that that was going to trend away in AI. Now there's excitement about this as a direction.
It shouldn't lead in a centralized direction because you will have many agents. And agents pushing against each other in a dynamic equilibrium. Offense-defense is a good example of this. Much rather intuitively have a trillion AIs of slightly superhuman intelligence join society than one AI of 1 trillion IQ join society. All of these things are things we have to push for. All of these things are—we have this incredible technology. It can be pointed and deployed in many ways. There are a lot of people who may have a different opinion about how it should be used, and what is the best version of the world. But this is the one that is thought to be right.
Regarding how the compute shortage is going, there are all these ways where things at various times along the curve are going to get somewhat out of hand. There will be moments, like with any new thing of this importance. There are going to be moments of irrational exuberance. There are going to be moments of bust. There are going to be moments of constraint, and glut, and up, and down.
The trend toward more compute continues without reversal. Demand for sufficiently capable models at sufficiently low price points moves in only one direction. OpenAI has made a chip achieving incredible performance in units of intelligence per watt. The company is learning to build its own data centers and has a robotics program directed at manufacturing many data centers.
There exists relatively little discussion about self-replicating data centers compared to RSI as an AI that does AI research. The concept involves using AI to control robots, run fabs, build turbines or solar panels, and construct data centers. This capability appears within reach. The goal is to produce lots of really cheap compute.
Data centers are not labor-bottlenecked but rather permitting-bottlenecked and component-bottlenecked. The solution to high prices is high prices. Memory, GPUs, and everything have high prices, yet solutions have not materialized. The invisible hand is failing in this domain. Memory makers historically invested $20 billion of CapEx to make $19 billion, experiencing booms and busts. Until recently, making memory was never a great business over a long period.
Memory producers are each going to make approximately $100 billion of free cash flow in a year. However, companies remember their history of near-bankruptcies from overbuilding fabs based on promises that conditions were different. This leads to conservative CapEx decisions. Intel makes chips and could potentially re-enter the memory game, following tradition of switching back and forth.
Local inference creates a global allocation problem. Having everybody possess a portion of the world's limited supply of DRAM sitting in their own computer, used only 5% of the time, represents poor utilization. Maximum utilization should occur in the cloud given the supply-constrained environment. Many people's intuition runs opposite to this principle.
Market responses to high prices appear across the supply chain in turbines, batteries, concrete, and steel. In components for industrial robots capable of making things faster, new actuator makers are emerging. OpenAI's own chip development represents one response to high prices and limited supply.
Chip design involves creating the file eventually sent to tape out, which is well understood and represents a great problem for AI. Lithography involves only one company on Earth that knows how to do it. Given limited global silicon capacity, shared interest exists in optimization.
OpenAI maintains a culture where people are willing to try using AI to do work better, even when they've done something the old-fashioned way for a long time. This applies to AI research, software engineering, procurement, and activities outside typical AI use cases.
Current capability allows requesting a 5% compute efficiency win at a high level. No one yet knows how to request discovery of a new architecture representing a leap forward comparable to the transformer. Missing elements include smarter models with better high-level planning and conceptual reasoning.
For a while, possibly a long while, limits may need to be agreed upon regarding how smart single models are allowed to become. Lack of progress would result from this decision rather than AI inability to conduct good research.
Using GPT-6 and unreleased versions to develop GPT-7 raises expectations significantly. The iteration loop involves asking what questions should be posed to GPT-7. Simply requesting RSI is insufficient, as is requesting a 5% compute efficiency win. Agent swarms represent one approach to maximizing output from models.
No codified OpenAI standard way exists for AI-enabled AI development. The process involves lots of experimentation. Token budgets represent one of the big unresolved questions for AI use in enterprises generally. No hard budgets exist for individual engineers currently.
Some companies previously displayed approximate meeting costs in Google Calendar, showing everyone's compensation required to attend. Paul Graham observed that people at companies possess incredible power to save $17 on weekly office snack budgets, while meetings can cost unbelievable amounts without adding value.
AI usage is not a company's biggest spend problem relative to other activities. Even if somewhat inefficient, AI spending is not where focus should be directed. Two years from now, companies may spend all of Stripe's revenue or profit on OpenAI services. Currently, tokens represent approximately 1% of Stripe's OpEx, which feels too low.
For many companies, AI spending will represent tens of percentage points of OpEx. The right number depends on company type. The cost per gigawatt of leading-edge AI should decrease because the fundamental cost of turning sand into intelligence should decline, as should energy costs, chip costs, and supply chain costs.
Supply must increase substantially to achieve cost reductions. A compute glut is desirable, producing more memory, data centers, and all inputs so anyone wanting a computer can have one. Current cost estimates range around $70 billion per gigawatt for five-year total cost of ownership. A reduction to $20 billion per gigawatt would be preferable.
Personal agents represent one of the most exciting developments in AI. Dots launched recently, alongside other entrants. The winning personal agent will feature intelligence, context, and tools. No one has yet created something people cannot imagine their life without, unlike coding agents or ChatGPT.
The computer use dynamic involves agents using computers. The world will eventually converge on allowing people to tell AI to use their computer on their behalf, including software. Consumer preference strongly favors existing primary chat apps over business-specific messaging interfaces.
Businesses resist allowing their services to be used inside other AI apps due to concerns about losing control and moats. However, defensibility derives from brand loyalty and product selection rather than HTML construction ability. Amazon maintains strong defensibility through its fulfillment network.
Any web form field represents fake work because people are not inventing information for the first time. Medical forms requiring repeated entry of family history exemplify this waste. Previous attempts at autofill plugins addressed this frustration clumsily.
Businesses remain attached to HTML and mobile apps as sources of business defensibility. This attachment may be transitory. The iPhone App Store represented the last similar phenomenon where companies felt compelled to build apps despite concerns about handing consumer relationships to Apple.
Benchmaxxing remains a concern. OpenAI endeavors not to benchmaxx. Two areas matter most: scientific discovery and doing things people cannot do, and being enjoyable to work with. No great benchmarks exist for either being enjoyable to work with or for the smartest scientific discovery capabilities.
The industry progressed through a glazing era, then a stroppy and ornery era where models disagreed and nitpicked on minor points. Current models are better than previous versions. Emmett Shear's desired model was a really thoughtful and challenging but still respectful senior coworker.
Tokens are disliked as a metric for pricing or measurement. The standard unit of AI should not be tokens. Outcomes also fail as a metric because they differ too much across use cases. A previous consideration involved pricing models without tokens, sending a price back after receiving a prompt, similar to AWS billing. Customer response was negative, with people preferring inflated tokens over uncertainty about long-term costs.
Nanometers represent a fake unit with little relation to reality in chip gate length measurements. Creating an intelligence equivalent token would similarly be made up and undesirable. Tokens are real in some sense but not a good unit.
Tokens that lack business value are considered fake from a commercial perspective. The primary defense of tokens is that wasted tokens should be fixed quickly rather than streaming noise instead of signal, as Claude Shannon would not approve. There exists a floor of the information theoretic most useful tokens on both sides.
Right now, the decision involves whether to use 10-cent-per-million tokens from a small model or $10 per token from a big model. The length of the chain of thought is likely to be a consideration, representing cognitive overhead that doesn't actually add value.
The question of whether chain of thought is included in the price charged is addressed. Most people include chain of thought in what they charge.
The Guinness was not as bad as expected. While not as good as an IPA, it was still considered good. The preference is that an IPA is great, but the Guinness is pretty good in the context of this podcast.
Keep Cheeky Pint in your library
Save the videos and channels worth coming back to, and find them again in one place.
