Why Traditional Benchmarks Fail Modern AI Models with OpenAI Research Scientist Noam Brown
In a Nutshell
Traditional benchmarks understate model capabilities by not accounting for test-time compute scaling. Modern models like o1.5 show substantial gains when evaluated across compute budgets, but these improvements are invisible in standard benchmark grids that lack a tokens/cost/time axis. Safety evaluations and responsible scaling policies remain misaligned with this reality, as they fail to test what models can achieve with large inference budgets, creating evaluation gaps that grow worse with each release cycle.
These notes were generated by AI and may contain inaccuracies.
With GPT-3, you couldn't scale test time compute. If you gave it a budget of $10 million and asked what GPT-3 could do, it really couldn't do that much more than what you could do with $10 or $1. The current frameworks and responsible scaling policies don't really account for the amount of test time compute. They just say what's the capability of the model. The problem is we're in a world now where the capability of the model is a function of how much money you put into it. If you give it a budget of $10,000, it can do a lot more than what it can do with a budget of $10. Give it a budget of $10 million, you can do even more. At what budget should you evaluate these models? The policies that exist today don't really address that question.
The motivation was the release of o1.5 and the initial reaction was skepticism that it was a substantially better model. That only lasted for a few hours before people had time to play around with it and saw that it was actually substantially better. A lot of the skepticism came from the benchmark grid that was published. Whenever a new model is released, there's this benchmark grid where they show all these different benchmarks on the x-axis and the performance of different models on the y-axis. If you look on paper at the difference between o1.5 and o1.4 or other models, it was an improvement, but it wasn't a huge improvement. It was only a few percentage points in some benchmarks.
The reason why it doesn't show up as so much better on the benchmarks is because the benchmarks are being presented in the wrong way. They're not controlling for the amount of test time compute that is being used on that benchmark question. It turned out that o1.5 is just much more efficient with its thinking. If you run it at max settings, o1.4 is thinking for a lot longer. It takes longer to get back a response than o1.5. Once you control for the amount of thinking time, you can see that o1.5 is a substantial jump over o1.4. That's people's day-to-day experience with it.
When asked why not just have o1.5 think for as long as o1.4, the question becomes how long should they think for? The typical response is until the performance plateaus. The point at which it plateaus is actually really far out these days. It was true in GPT-3 land back in 2022. The models couldn't really think productively for that long. You could just run them until they plateau. It's not that far away. But what we're seeing today with modern models is that o1.5 and other models can think for weeks even before having performance plateau on some of these benchmarks. The point at which they plateau is simply too far out to reasonably test. We all need to actually enforce either a patience limit or a budget limit from a token perspective now, and that wasn't true a few years ago.
The proper way to evaluate the models now is to either have some kind of budget for the benchmark, whether it's tokens or cost or time or whatever, or plot the performance as a function of the amount of test time compute that's going into the model. Then it becomes much more clear how to compare the performance between these different models.
For things like cyber, the models continue to improve at 100 million tokens. If you run them for 100 million tokens, they're still improving beyond that point. The performance is not just a discontinuous jump. You can see the slope of improvement over those 100 million tokens. You could do some kind of evaluation up to a certain budget and then project what the performance would look like. There hasn't been a lot of research on this yet. This would be a great paper to publish. Can you predict what the performance looks like at an inference budget of $10,000 only using inference budgets up to $10 or $100?
The idea that models just think for a week and then respond sounds nice and the benchmarks look great, but it's not very practical when working because you ask the model a question and then sit there for a week waiting for it to come back. What people have found most effective is to iterate quickly with the models. The thinking time needs to be flexible. When it makes sense to respond quickly to the user, it should respond quickly. When it makes sense to think for a long time and the user wants it to think for a long time, then it makes sense to think for a long time. People have been striking the right balance given what they have to deal with right now.
Benchmark maxing is also motivation for writing the essay. It's really easy to show much better performance than previous models on benchmarks by scaffolding a bunch of models together. If you run the model five times and take the best of the five responses, or ask a judge which one it thinks is best, you can get much higher scores than that model. It's really easy to make something that looks a lot better on paper but is actually not better once you control for the amount of test time compute. That's one concern when it comes to benchmark maxing. It's a little misleading.
There's always a risk of just optimizing for the benchmark. At OpenAI they're pretty good about not trying to optimize for specific benchmarks. But once you put out a benchmark, it's always at risk of just being optimized for. One way to address that is to keep a held out private set that isn't publicly available.
Everybody has their own set of questions that they like to ask the model whenever it comes out. For Noam Brown lately, it's been using models to make poker bots and see how good they can make a poker bot. It's a nice eval because there is very little open source code for making poker bots. There's a lot of published papers on it, but you really have to reason through everything. It requires a lot of reasoning and iteration and a lot of small gotchas that he can see where the models fail along the way.
When the early models were really bad at it, they could not basically do anything. With o1.2, he was able to work with it to make a river solver. That's the final stage of poker. That itself was really impressive. He had to work with it a little bit, but he was able to make the river solver probably about five times faster than he would have alone. Blockers was always a big issue. With a bit of gentle steering, it felt like a grad student. They would run into issues, but he would know what those issues were and know how to fix them. He could make suggestions and it would go off and come back with something really good.
The optimization was very impressive. It was able to make it 10 times faster than what he was able to do because it optimized the code so well. The downsides with o1.2 were that it was gaslighting him a lot. He always had to be very careful checking it and making sure it was actually doing what it said it did. There was a point where as a unit test he told the model he had $100 in the pot and folded, and asked how much he was losing. The model said $92. When he said that's crazy, the model said it's close to 100, it's fine. The models had this problem where they would gaslight you a lot.
Once they got to o1.5, it was way better. It was able to basically do it zero shot. He's been working on just doing a full scale poker solver and it's basically able to do the whole thing with some gentle steering. He wouldn't be surprised if in 6 months or a year from now, the model is able to do zero shot an entire poker solver, basically his entire PhD thesis in one go.
The safety evaluations thing is a bit of an inconvenient truth. A lot of the labs have responsible scaling policies and preparedness frameworks. The idea is that whenever a model is released, they go through a series of evaluations to measure if there are dangerous capabilities. Could these models do things that we wouldn't want a bad actor to do? If the model isn't very capable, then it's no big deal. But if it is very capable, if it could be used to make bioweapons, then you want to put in mitigations against that.
The question is how do you evaluate whether the model is capable of that? They have various protocols about how they do these evaluations. But a lot of these frameworks were developed around the era of ChatGPT when test time compute scaling was not really as much of a thing. With GPT-3, you couldn't scale test time compute. If you gave it a budget of $10 million and said let's see what GPT-3 can do, it really couldn't do that much more than what you could do with $10 or $1. The preparedness frameworks and responsible scaling policies don't really account for the amount of test time compute. They just say what's the capability of the model. The problem is we're in a world now where the capability of the model is a function of how much money you put into it. If you give it a budget of $10,000, it can do a lot more than what it can do with a budget of $10. If you give it a budget of $10 million, it can do even more. At what budget should you evaluate these models? The policies that exist today don't really address that question. Some do better than others, but for the most part this is not really a factor that's being heavily considered.
It's the mirror image of the capability question. If the models can continue to do more and more without asymptoting on some tasks at very large budgets, then they should also be able to do so for tasks we don't want them to do as a society. Testing for that and what budget is allocated also seems out of sync from the model release cycle itself. There's been this acceleration where you get a new model every few days and weeks at this point versus 6 months. There's a line in the essay where it says the only way to truly evaluate an agent on some very long-running task might be to run it for a year. That's going to be true of both useful and negative tasks.
As the models have become stronger, they're better able to operate over longer horizons. With GPT-3, if you wanted to run it for a week, there's really not much you could do to scaffold it into something useful that could actually run for a week. But with the most recent models, you can actually scaffold o1.5 into doing a series of experiments that can run for weeks, for months.
He hasn't really scaffolded something together where he just tells it to run for weeks. He thinks at this point it could 100% do the river solver if he just gives it a goal. He doesn't think it's at the level yet where it could do the full poker solver if he gave it a goal and told it to go run for a month. But we're going to pretty soon be at that point where he probably could just tell it to go work on this for a month and then come back with a full complete poker solver that's state-of-the-art.
The problem is if you want to evaluate the capabilities of a model, what it can do after running for a month, the only way to be fully sure is to actually run it for a month. If you want to know after 6 months, the only way to know fully is to run it for six months. The model release cycle is that they're releasing new models every two or three months at this point. A model comes out, it takes two or three months to push it to its limits and then another model comes out. Nobody actually knows what the ceiling of capabilities are for these models because nobody's actually run them for long enough to really tell.
When o1 came out, people started running things that took over a week for it to finish. People actually didn't realize that this was a big deal until after a week after it was released. The implications are interesting because what do the labs do to fully evaluate their models before they're released? It's actually very difficult because the only way to really do the evaluations is to delay the model release cycle. There's a lot of competitive pressure right now to not do that.
There is exciting latent capability in the models that are already released that people have not fully explored. A really great example is the Erdős unit distance problem. They used an internal model at OpenAI a few weeks ago to disprove the Erdős unit distance conjecture. This was a pretty big deal in the math community. It was the first problem that a lot of mathematicians had really spent a lot of time on, and the model was able to do something that they weren't able to do in a way that was actually interesting and useful for mathematicians. It did it at a budget that was dirt cheap. They didn't put a lot of effort into this. They just trained a new model and were curious what it could do. They ran it through some problems. This one at a pretty low budget, it said it had a disproof. They were able to verify that the proof is correct.
After they announced the results, a bunch of people found that you could get the answer out of o1.5 as well. It's not as simple as just asking o1.5 what's the disproof. You had to scaffold it a bit. You had to steer it a bit. Somebody found you ask o1.5 to list a bunch of ways that you could tackle this problem. It lists one of the paths that are actually promising to get to the disproof. Then you tell it to explore this some more. If you do this enough times, it actually ends up arriving at the disproof. What this means is you could in principle ask o1.5 as a general purpose scaffold to list a bunch of different strategies and then for each strategy tell it to investigate that strategy, and it would probably be able to arrive at the disproof with a general purpose scaffold. That scaffold would be very expensive. It would probably cost $1,000 to $10,000 to $100,000. But it would be possible and it would have been possible for somebody to disprove the Erdős unit distance conjecture before they did using a general purpose model. Nobody had explored sufficiently what happens if you put $100,000 worth of compute into o1.5. The answer is you probably could get stuff like that out of it. People should be experimenting more with the current generation.
The model release cycle is every couple months we put out a new model that's even more powerful, so the cost of disproving the Erdős unit distance conjecture drops by 10 or 100x with every model release cycle, probably in some cases more. There's the meme that you shouldn't bother doing any engineering work when you should just wait for the next model, go on vacation, come back two months later and it's a thousand times cheaper.
At OpenAI, they're in a period where progress is very fast and the models are becoming more capable. One of the things that they're actively not doing is spending all their time just going through all the mathematical open problems and physics problems and pushing the models to their limits to see what they can prove or disprove. The focus should be on how do we make even more capable models? How can we get them out safely to the world as quickly as possible so that all the scientists in the world can use these models to solve the problems themselves. It's really tempting to just put all efforts into scaling up these models and see what they can do at their limits right now, but really the focus should be on how do we use these models to make even more powerful models that can do everything much more cost-effectively.
He doesn't think we're at the point where you just give it an arbitrary extremely high inference budget and it's just super intelligent across the board. There are some benchmarks where the models will just not improve if they have more inference budget. A lot of factual retrieval kind of questions fall into this category. If you ask a person when was Abraham Lincoln born and they don't know the date, they could sit there and think about it for a week. If they don't have access to Wikipedia or something, they're not going to be able to do better answering that question if they thought about it for a week compared to 5 seconds.
Some benchmarks clearly improve with additional test time compute while others do not. Factual retrieval questions show improvement when models are given more time to think, but extended time does not enhance performance on tasks like remembering dates. Sudoku represents the opposite extreme where models can improve without limit given more test time compute through a simple strategy of trying random number combinations until constraints are satisfied.
All benchmarks exist on a spectrum between these two extremes. Models are not currently at a level where unlimited test time compute would enable them to perform all human jobs, since some benchmarks show no improvement regardless of additional compute.
Current models lack strong research taste, making them effective complements to human researchers rather than replacements. Models can optimize existing algorithms developed during Noam Brown's PhD work, achieving 10-100x speed improvements, but cannot generate novel algorithms that surpass published work even with substantial time allocation.
Research taste is expected to improve with each model release cycle, following patterns seen in coding and mathematics where sudden inflection points led to practical usefulness. The current state accelerates specific research activities while creating bottlenecks around tasks that do not see similar acceleration.
Fast takeoff is relative to current rapid progress, but overnight intelligence explosions where models achieve superhuman capabilities through self-improvement breakthroughs are not anticipated. Models depend heavily on large-scale test time compute to reach peak intelligence, creating time bottlenecks that prevent instantaneous capability explosions.
Time itself functions as the primary constraint across frontier labs, with researchers working intensive hours due to awareness of capability overhangs and the need to advance quickly.
Multi-agent approaches are considered well-explored but require frontier-scale models to fully unlock capabilities. Current implementations only scratch the surface of potential applications. Human civilization demonstrates how accumulated knowledge across billions of individuals over time enables greater achievements without biological intelligence increases, a dynamic not yet replicated in AI systems.
AI models currently operate with short context windows and disappear after use, limiting knowledge sharing and compounding. Early systems like Multibook and Open Claw, despite initial overhype, indicated directions toward coordinated, compounding AI states where models share and build upon knowledge globally.
Competition among frontier labs remains intense, with current models accelerating researcher capabilities within limits. All frontier labs recognize the high stakes and dual potential for beneficial or harmful outcomes, creating awareness that extends beyond pure competitive dynamics.
Models have advanced sufficiently for high-stakes applications where outputs can be trusted more than expert human responses. Practical uses include tax advice and real estate paperwork guidance during condo purchases. Earlier skepticism from 2022-2023 experimentation periods no longer applies given current reliability levels.
Publishing benchmarks without an x-axis representing tokens, cost, or time creates a suboptimal equilibrium where researchers recognize the issue but continue existing practices due to expectations. The goal is establishing new norms where companies feel comfortable publishing results with proper scaling axes rather than default grid formats.
Routing approaches that direct tasks to different models based on complexity requirements must be evaluated against equivalent test time compute allocations. Consensus methods among multiple models can improve performance, but require comparison against single models given equivalent compute budgets to determine actual advantages.
Real-world performance gains from routing may not match benchmark improvements, requiring skepticism about whether routing delivers significant advantages or merely optimizes for specific evaluation metrics without meaningful practical benefits.
Keep No Priors: AI, Machine Learning, Tech, & Startups in your library
Save the videos and channels worth coming back to, and find them again in one place.
