Michael Nielsen – How science actually progresses
In a Nutshell
Michael Nielsen explains scientific progress as messy and non-linear, challenging myths like instant falsification (e.g., Michelson-Morley didn't immediately kill ether theories; relativity won via unification and later muon experiments) and long verification loops (e.g., heliocentrism waited 1800 years for parallax; Darwin needed deep time from geology). He critiques AI's role, praising AlphaFold's data-driven success but warning it fits models without deep explanations, risks overfitting without principles, and struggles with paradigm shifts amid diverse, path-dependent tech trees where new fields continually emerge to counter diminishing returns. Key implications: maintain diverse research programs, value stuck time for deep learning, and anticipate vast inter-civilizational trade from non-convergent discoveries.
These notes were generated by AI and may contain inaccuracies.
Michael Nielsen is a pioneer of quantum computing, wrote the main textbook in the field, contributed to the open science movement, authored a book on deep learning credited by Chris Olah and Greg Brockman for getting them into the field, and is currently a research fellow at the Astera Institute writing a book about religion, science, and technology. The discussion focuses on recognizing scientific progress, particularly relevant for AI efforts to close the RL verification loop on scientific discovery.
Michelson-Morley experiment (1880s, first in 1881, famous version 1887) tested theories of the ether, introduced by Robert Boyle in the 1600s. Ether was hypothesized as the medium for light propagation after Boyle showed light travels through vacuum unlike sound. The experiment aimed to detect an ether wind from Earth's motion through the ether using interference: light parallel to ether wind accelerated, opposite slowed. Result: no ether wind detected, ruling out some ether theories but not all. Michelson continued believing in ether until his death in late 1920s, conducting experiments into the 1920s. Physicist Miller in 1920s at Mount Wilson claimed to measure ether wind at high altitude.
Einstein later stated uncertainty about knowing the Michelson-Morley paper, though evidence suggests he was aware; it was not central to his thinking. Common narrative (e.g., YouTube) portrays it as proving no ether, creating crisis solved by special relativity. Reality: experiment distinguished ether theories (e.g., stationary vs. Earth-dragged ether). Post-1887, leading physicists accepted no ether wind but developed other ether models. Michelson redid 1881 experiment in 1887 after critiques (e.g., Rayleigh).
"Subtle is the Lord, but malicious He is not." (Einstein on Miller's results)
History from Abraham Pais's Subtle is the Lord and Imre Lakatos's The Methodology of Scientific Research Programmes. Falsification not straightforward: unclear what is falsified (specific ether versions?). Does not induce special relativity. Leading physicists responded by refining ether theories, not abandoning ether. Word "the ether" misnomer; many competing theories.
Lorentz (end 19th century, pre-Einstein) developed Lorentz transformations for reference frames, interpreting length contraction and time dilation as ether effects (pressure warping clocks/lengths). Experimentally indistinguishable from special relativity. Introduced local time (mathematical, later Einstein's time in inertial frames). Poincaré closer to physical clock time.
Muon experiments (~1940, published ~1941): cosmic ray muons decay slowly, matching special relativity time dilation predictions, not classical theory. Would challenge Lorentz's ether if he lived (died ~1930). Multiple similar experiments. Community adopted relativity interpretation before full experimental distinction.
No centralized authority or standard procedure. Great scientists (Lorentz, Poincaré, Michelson) remained unconvinced post-community shift. Poincaré grasped principle of relativity and constant light speed in inertial frames but saw length contraction as dynamical (external force), not kinematic rethink of space-time (1909 paper). Einstein simplified by rethinking space-time fundamentals.
Einstein as teenager in 1890s believed in ether but less attached than experts. Expertise can hinder letting go.
Aristarchus (2nd century BC) proposed heliocentrism; Athenians dismissed due to no observed stellar parallax (measured 1838). Copernicus not more accurate or simpler than Ptolemaic model (Copernicus added epicycles for perfect circles). Newton unified celestial (Kepler's laws), terrestrial parabolas, tides via gravitation—compelling unification.
From Keynes essay: Newton "not the first of the age of reason... the last of the magicians." Applied same method to alchemy, theology, math (Principia, 1687). Careful, sober even in esoteric works (composed same 25 years as math studies).
Parsimony, aesthetics drove shifts (Einstein rejecting ether, Newton). Heuristics may not generalize across time/disciplines; bottlenecks occur where prior methods fail. Science not reducible to crank-turning process; needs diverse ideas. Quantum mechanics, evolution shocking ideas.
Principia Mathematica (1687), Origin of Species (1859). Darwinism conceptually easier yet later. Thomas Huxley: "How extremely stupid to not have thought of this." Nobody says that of Principia.
Artificial selection known to breeders. Darwin's genius: centrality to biology, explaining variety across biosphere, linking to geology. Origin compiles evidence/examples. No single overwhelming proof; no genes understood. Alfred Wallace independently co-discovered ~1850s; Wallace sent manuscript to Darwin, joint presentation.
Lucretius (1st century Roman): species fitted via one-time past generative filter, not ongoing gradual process or tree of life (all life common ancestor, unsurprising given origin bottleneck). Darwin compiled cumulative evidence; Newton had quick verifications (Earth fall, Moon orbit, tides).
Why was the period in the 1850s or 1860s the right time for these ideas to form? One factor is geology: in the 1830s, Charles Lyell figured out that Earth has existed for millions and billions of years. Paleontology shows fossils have existed for that entire time, with life going back a long way. Fossils even reveal intermediate species showing the tree of life, including between humans and other apes.
The age of colonization enabled voyages doing biogeography, which must have been necessary.
There's a huge history of parallel innovation and discovery in science. If an idea isn't discovered for a long time and then spontaneously many people come up with it, that shows the building blocks were necessary. Lyell and other geologists' idea of deep time in the early 1800s was crucial; Darwin was very influenced by Lyell. Without tens or hundreds of millions of years, evolution looks like a non-starter—on a 5,000 to 10,000 year timescale like Bishop Ussher's, it would require massive rates during human lifetimes, which we don't see.
An example giving pause on the verification loop in AI is AlphaFold's big success, which really isn't about AI. A massive fraction of the success is the Protein Data Bank, X-ray diffraction, NMR, cryo-EM, and several billion dollars spent obtaining 180,000-odd protein structures. It's decades of experimental data acquisition, with fitting a model as a tiny fraction of the investment.
The AI bit is impressive but a small part of the total story.
Philosophically, is AlphaFold a scientific theory or explanation? The world is becoming harder to understand in some domains, needing models fitted to data rather than underlying principles explaining broad phenomena. Compare general relativity (GR)—just equations—with AlphaFold's 100 million uninterpretable parameters encoding relationships. GR predicts unanticipated things like Mercury's orbit precession; AlphaFold lacks that explanatory reach.
Three answers:
-
Conservative view: Want deep explanatory principles with few free parameters and simple models explaining a lot. AlphaFold doesn't fit; it's a helpful model but not a scientific explanation.
-
Maybe it contains lots of little explanations inside. Interpretability work extracts circuits doing interesting things, like in chess models such as AlphaZero. Strategies from AlphaZero were borrowed by Magnus Carlsen after public forensics; experts noticed he changed his game radically. Humans extract meaning, viewing models as a source of explanations—not legible up front, but extractable.
-
New type of object to take seriously as explanations. Now we can merge and distill them. Historically, a 100-page equation in 1920 was unworkable, but tools like Mathematica enable working with them, sometimes yielding simple answers. AlphaFold-like models could be treated similarly—interesting operations possible, even if not classic explanations.
Suppose in 1500 we developed deep learning before cosmology. Training on stars and planets' weird behaviors, interp would just add epicycles like Ptolemy's model—parameters encoding more epicycles. It took one mind to integrate into a globally better theory like Copernicus. We don't have the verbs yet for these models.
Apply constraints like simplest explanation or 90/10 version, distilling to parsimony. But raw gradient descent won't switch paradigms like Ptolemy to Copernicus or Newtonian gravity to GR—local accuracy worsens globally before improving.
Einstein developed special relativity, realizing Newtonian gravity allows faster-than-light signaling and backwards time travel via action at a distance. This forcing function demanded a new theory. Simplest fixes failed, leading through ugly intermediates to GR's simple beauty.
For AI accelerating science:
- Well-understood domains (e.g., protein folding): Train raw model on data via gradient descent.
- Fundamental discoveries like GR: Couldn't train on all observations. Required decades, independent research programs with biases (e.g., Einstein's equivalence principle thought experiment). Keep diverse programs alive; verification loops long.
Same answer correct in some cases, wrong in others. Uranus's orbit anomaly predicted Neptune—success for Newtonian gravity (Le Verrier, 1846). Mercury's 43 arcseconds/century precession led to Vulcan search inside its orbit—no planet found. Similar ideas succeed or fail; pursue both.
Illustrates falsificationism's difficulty: Anomalies like Uranus seem to falsify but are saved by ancillaries (e.g., unseen planet). For Mercury, ad hocs like cosmic dust, small planet, magnetic fields persisted. Pioneer spacecraft anomaly (1990s) excited new gravity theories but was thermal radiation asymmetry.
Exceptions everywhere; 99.9% mundane. No ex ante heuristic distinguishes; science full of them. History told simply ignores selection bias—Nobel for right exception, but rare.
AI good at tight verification (coding unit tests, experiments), but infinite theories fit any experiment. Latching onto the right one retrospectively hard. Lakatos examples: Prout (1815) hypothesized atomic nuclei have whole number weights (made of hydrogen); most elements fit, chlorine 35.5 exception led to ad hocs (impurities, fractions). Chlorine measured 35.46. Isotopes (physically, not chemically separable) explained it—85 years later.
Verification hostile to correct theory; defend remnants, integrate anomalies.
Bottlenecks multiple. AlphaFold shocked structural biologists, speeding one type. But not all.
Even coding: Programmers excited, bottleneck shifts to interesting design ideas—no verification loop. Previously, coding time allowed idea generation; now prototypes in 3 hours, but ideas lacking.
By 2028, AI preps taxes like competent GM. Already close: Used LLM on 2025 transactions via Mercury's MCP (detailed info: notes, memos, JPEGs, PDFs). Flagged personal vs. business expenses well. Bay Padel charge seemed personal, but receipt/note showed team bonding at retreat—legit business.
Mercury exposes this; traditional banks lag. Mercury: fintech, not FDIC-insured bank; services via Choice Financial Group, Column N.A., FDIC members.
If we met aliens, they might have totally different technological stack. Challenges assumption science is early civilization milestone, converging universally after basics.
Tech/science tree larger than realized. Theory of everything doesn't end physics. Computer science: 1930s Turing/Church computation theory; 90 years exploring consequences—ideas like public-key cryptography deep, non-obvious, hidden then.
Different exploration paths; we're low down.
Phases of matter: School taught 3-5; now superconductors, superfluids, Bose-Einstein condensates, quantum Hall, fractional quantum Hall—many more, designable within physics laws. Bottom of tree.
Programming: Ludicrous we've found all deep ideas; keep discovering. Humans slow (jumped-up chimpanzees); million years ahead, vast ideas.
Knuth preface to The Art of Computer Programming (1960s start): Mathematician said come back with 1,000 deep theorems; decades later, Knuth: clearly are.
Long-term: Choices branch tree. Visual bias vs. aural (animals); exotic civilizations' perceptions change exploration.
This is such an interesting take. I want to better understand it.
One way to understand it is that there might be some things which are so fundamental and have such a wide collision area against reality that they're inevitably going to discover, like general relativity. Numbers. Numbers. Of all the intelligences in the Milky Way galaxy… Maybe that number is one. Well, actually, arguably we've already increased the number. But of all of those, what fraction have the concept of counting? It does seem very natural. What fraction have discovered the idea of some kind of decimal place system? Interesting question. Maybe we're missing something really simple and obvious that's actually way better than that. What fraction got there immediately? What fraction had to go through some other intermediate state? What fraction uses linear representations versus a two-dimensional or a three-dimensional representation? I think the answers to these questions are just not at all obvious. It's a lot of design freedom.
On theoretical computer science, this is going to be extremely naive and arrogant, but I took Scott Aaronson's class on complexity theory, and I was by far the worst student he's ever had. What I remember is there was this period, in which you were one of the pioneers, where we figured out the class of problems that quantum computers can solve and how it relates to problems that classical computers can solve. It was groundbreaking. It's crazy that this works. Since then… There's literally this website called Complexity Zoo which lists out all the complexity classes. If you have this complexity class with this kind of oracle, it's equivalent to this other class. It feels like we're building out that taxonomy.
There are a couple ways to understand what you're saying. One, maybe you disagree with me that this is actually what's happened with this field. Another is that while that might happen to any one field, who would've thought in 1880 that computer science, other than Babbage, was going to be a thing in the first place? We're underestimating how many more fields there could be. Or maybe you think both, or maybe a third secret thing. A very common argument here is the low-hanging fruit argument. The argument that says there should be diminishing returns. In fact, empirically we see this. The amount of scientists in the world has exponentially increased. I think it's worth thinking about why you expect diminishing returns and how well that argument actually applies in practice.
Dessert Buffet Analogy
An analogy I like is thinking about going to an event, like a wedding, and you go to the dessert buffet. They've put out thirty desserts. Naturally, what people do is the best desserts go first. We don't quite have a well-ordered preference there, so maybe there's some difference, but human beings are fairly similar, so the best desserts will go first. This is an argument for why you expect diminishing returns in a lot of different fields. If it's relatively easy to see what's available and people have similar preferences, then the best stuff goes first and it just gets worse and worse after that.
If you look at a very static snapshot in time of scientific progress, maybe there's some truth to that. But if somebody is standing behind the dessert table and is replenishing and restocking the desserts and keeps adding new ones in, it may turn out that a little bit later, much better desserts appear, and you're going to go and eat those instead. Scientific progress has a little bit of that flavor.
Emergence of New Fields
We go through these funny time periods. Computer science is a great example, where computer science basically arose as a side effect of some pretty abstruse questions in the philosophy of mathematics and logic. You've got these people trying to attack these rather esoteric questions that seem quite high up in exploration, and they discover this fundamental new field, and all of a sudden there's an explosion there. The diminishing returns argument just didn't apply there. We just weren't able to see what was there. This has been the case over and over again. New fields arrive and all of a sudden, and boom, it's easy to make progress again. Young people flood in because you can be twenty-one and make major breakthroughs rather than having to spend twenty-five years mastering everything that's been done before. It's obviously very attractive. I'm not sure anybody understands very well the dynamics of that, or how to think about why the structure of knowledge is that way, where these new fields keep opening up. But it does seem empirically to be the case.
Despite the fact that that is the case… Take deep learning. Obviously, this is an example of a new field where twenty-one-year-olds can make progress and it's relatively new. Fifteen years or so since it got back into high gear. But already we're in a stage where you need billions, tens of billions, or hundreds of billions of dollars to keep making progress at the frontier. There are a couple ways to understand that. One is that it actually is harder than the kinds of things the ancients had to do, or is more intensive at least. Second is it might not have been, but because our civilizational resources are so large, the amount of people is so large, the amount of money is so large, we can basically make the kind of progress it would have taken the ancients forever to make almost immediately. We notice something is productive and immediately dump in all the resources. But it's also weird that there's not that many of them. I feel like deep learning is notable because it is one big exception to the fact that it's hard to think of other examples. I think that's a consequence of the architecture of attention.
Centralized Attention and Fashion
At any given time, there's always a most successful thing. If deep learning wasn't a thing, maybe you'd be talking about CRISPR. Maybe we wouldn't think about solving the protein structure prediction problem as a success of AI. Maybe we would have figured out how to do it with curve fitting, more broadly construed, and we'd just be like, "Wow, that took a lot of computing resources." But protein structure prediction might be an enormously important thing. There is always our biggest thing. What you're pointing at is more a consequence of the way in which attention gets centralized. It's basically fashion, is what I'm saying. It's not just fashion, but there is some dynamic there.
There's a very interesting and important implication of this idea. That the branching is so wide and so contingent and so path-dependent that different civilizations would stumble on entirely different technology stacks. There's a very interesting implication that there will be gains from trade into the far, far future, which might actually be one of the most important facts about the far future in terms of how civilizations are set up, how they coordinate, and how they interface. There's not this "go forth and exploit." There are humongous gains to trade from adjacent colonies or whatever.
Barriers to Trade and Comparative Advantage
There's a question of what's actually hard. If it's just the ideas, well, those spread relatively quickly. It's relatively easy to share ideas. If it's something more, it's almost a Dan Wang kind of idea where there's some notion of capacity. You need all the right techs, you need all the right manufacturing capacity, and so on. So civilization A has a very different kind of manufacturing capacity, and it's just not so easy to build in civilization B. Even if civilization B is ahead, I think that becomes true. There is a comparative advantage which is going to provide massive benefits to trade in both directions. Eventually, you expect some diffusion of innovation. It is funny to think about what the barriers are there.
GitHub for Aliens and Biological Libraries
A fun thought experiment I like to think about is GitHub but for aliens. Somebody presents you with all of the code from some alien civilization. I don't even know what code means there, but their specification of algorithms. It would have many interesting new ideas in there, and it would take forever for human beings to dig through and try and extract all of those. The origin of this for me was thinking about proteins in nature. We've been gifted this incredible variety of machines which we don't really understand at all. We just have to go and try and understand them on a one-by-one basis. We're still understanding hemoglobin and insulin and things like this. There are hundreds of millions of proteins known. So it is a little bit like that. We've been gifted by biology this immense library of machines, no doubt containing an enormous number of very interesting ideas, and we're just at the very, very beginning of understanding it.
I suppose your point—I need to relabel your argument slightly—but you think of that as a gift from an alien civilization, which obviously it isn't, but you think of it that way. And oh my goodness, there's so much in there and we're going to study it. Goodness knows how long we could continue to study it. There are tens of thousands of papers about hemoglobin and things like that, and we still don't understand them, and yet we're getting so much out of it. Just think about insulin alone. It's such an important thing.
Earth's Biology as Alien Tech
That's an incredibly useful intuition pump, that you have on Earth… I had Nick Lane on where he had this theory about how life emerged, but whatever theory you have, something like DNA has had four billion years. You have an alien civilization come here and be like, "There's all these interesting things to learn about material science." Think about kinesin walking along. We know almost nothing about these proteins, and yet the tiny few facts we do know are just incredible. The ribosome is another example, this miraculous sort of device, a little factory. All seeded by this particular chemistry on Earth with nucleic acids and carbon-based life forms. That chemistry gives rise to all of these interesting things which an alien civilization would find very interesting. That very seed, which must be one among trillions of possible seeds of general intellectual ideas, leads to all this fecundity. That's a very interesting intuition pump.
I want to meditate on this "gains from trade" thing because I feel like there's something very interesting about this idea that if you have this vision of how technology progresses and how it may be different in different civilizations, it actually has important implications about how different civilizations might interact with each other. The fact that there are going to be these huge gains from trade. It makes friendliness much more rewarding?
Yes. That's a very important observation. I hadn't thought about that at all. That is a very interesting observation. It is funny. Comparative advantage is something that people love to invoke and it's a very beautiful idea obviously. There are limits to it. It's a special limited model. Chimpanzees can do interesting things, but we don't trade with them. I think it's interesting to think about the reasons why.
Power Imbalances and Transaction Costs
Part of it is just power, I think. Once there's a sufficiently large power imbalance, very often—not always, but very often—groups of people seem to shift into this other mode where they just seek to dominate. Maybe there's something special about human beings, but maybe it's also a more general thing. You need all these special things to be true before groups will trade. It's not necessarily obvious.
I think the big thing going on here is one, transaction costs. Two, comparative advantage does not tell you that the terms on which the trade happens are above subsistence for any given producer. People often bring this up in the context of, "Well, humans will be employed even in a post-AGI world because of comparative advantage." There are five different ways that argument breaks down, but the easiest way to understand it is: why don't we have horses all around on the roads? Because there's some comparative advantage between cars and horses.
- One, there are huge transaction costs to building roads that are compatible with horses and cars at the same time. In a similar way, AI thinking at 1,000 times the speed that can shoot their latent states at each other is going to find it way more costly than the benefit, in terms of interacting with a human being in the supply chain.
- Second, just because horses have a comparative advantage mathematically does not mean that it is worth paying $100,000 a year, or whatever it costs to sustain a horse in San Francisco. That subsistence isn't going to be worth the benefit you get out of the horse.
I do think it's interesting, the sheer fact… My expectation and my intuition obviously differs a great deal from yours on this. Most parts of the tech tree are never going to be explored. There are just too many interesting ways of combining things. There are too many deep ideas waiting to be discovered, and not only we, but nobody ever is going to discover most of them. So choices about how to do the exploration actually matter quite a bit. It's something I really dislike about technological determinist arguments. I'm willing to buy it low enough down when progress is relatively simple. But higher up, you start to get to shape the way in which you do the exploration. And it's interesting, we are starting to shape it in interesting ways.
Shaping the Tech Tree
There are various technologies that have been essentially banned. You think about DDT, chlorofluorocarbons, restrictions on the use of nuclear weapons, the Nuclear Non-Proliferation Treaty. Those kinds of things weren't done before the fact, but they're starting to get pretty close in some cases, where we just preemptively decide, "Oh, we're not going to go down that path." So that starts to look like a set of institutions where we are actually influencing how we explore the tech tree.
On where you would see these gains from trade, obviously you'd see the most where it's pure information that could be sent back and forth, because the information has this quality where it is expensive to produce, but cheap to verify and cheap to send. It'll be interesting how much of future productivity can be distilled down to information. Right now, it's hard to do. If China's really good at manufacturing something, there's this process knowledge that's in the heads of 100 million people involved in the manufacturing sector in China. But in the future, it might be easier if AIs are doing it. The question is to what extent our fabrication gets very uniform and gets really commoditized.
3D printers have been the next big thing for at least 20 years now. Why do they still not work all that well? Why are they still not at the center of manufacturing, and what comes after that? It is funny to look at the ribosome by contrast, which really is at the center of biology in a whole lot of really interesting ways. Whether or not that's the future of manufacturing is something very simple, where everything goes as throughput through a bioreactor or something like that. You send the information, and then you grow stuff, or you have some 3D printer that actually works. If they're good enough, then it does become much more a pure information problem, and some of this process knowledge becomes much less important.
Jane Street has a lot of compute, but GPUs are very expensive, and so even optimizations that have a relatively small effect on GPU utilization are still extremely valuable. Two of Jane Street's ML engineers, Corwin and Sylvain, walked through some of their optimization workflows at GTC. You're not bottlenecked on the network being too slow, you're bottlenecked on waiting for a different rank in your training not having completed the work. They talked about how Jane Street profiles traces and diagnoses bottlenecks, and then how they solve them using techniques like CUDA graphs and CUDA streams and custom kernels. With these sorts of optimizations, Corwin and Sylvain were able to get their training steps down from 400 milliseconds to 375 milliseconds each. This 25 millisecond difference might sound small, but given the size of Jane Street's fleet, that improvement could free up thousands of B200s. Jane Street open sourced all the relevant code. If you want to check it out, I've linked the GitHub repo and the talk in the description below. And if you find this stuff exciting, Jane Street is hiring researchers and engineers. Go to janestreet.com/dwarkesh to learn more.
Can I ask a very clumsily phrased question? There are these deep principles that we've discovered a couple of. One is this idea that if there's a symmetry across a dimension, it corresponds to a conserved quantity. It's a very deep idea. There's another—which you've written a lot about, written a textbook about in fact—about ways to understand what kinds of things you can compute, what kinds of physical systems you can understand with other physical systems, what a universal computer looks like, et cetera. Is your view that if you go down to this level of idea of Noether's theorem or the Church-Turing principle, that there's an infinite number of extremely deep such principles? Because I feel what makes them special is that they themselves encompass so many different possible ways the world could be. But no, the world has to be compatible with a couple of these very deep principles.
I don't know. All I have here is speculation and instinct. My instinct is that we keep finding very fundamental new things. It was quite formative for me to understand, as I gave the example before, these wonderful ideas of Church and Turing and these other people about universal programmable devices. Then you understand later, this also contains within it the ideas of public-key cryptography. Then you understand later, that also contains within it the ideas people refer to as cryptocurrency. There's a very deep set of ideas there about the ability to collectively maintain an agreed-upon ledger, which is built upon this. It's taken many years to figure out the right canonical form of those. Just this fact that you keep finding what seem like deep new fundamental primitives has been a very important intuition pump for me. I've given that particular example, but I think you see that same pattern in a lot of different areas.
What is your interpretation then of this empirical phenomenon where whatever input you consider into the scientific process or technological progress… Economists have studied this a million ways. It just seems to require a very consistent rate of X percent more researchers per year. There's this famous paper from a couple years ago by Nicholas Bloom and others where they say, "How many people are working in the semiconductor industry, and how has it increased over time through the history of Moore's law?" I think they find that Moore's law means transistor density increases 40% a year, but to keep that going the number of scientists has increased 9% a year, in the semiconductor industry. They go through industry after industry with this observation.
Is your view that there are these deep ideas, but they keep getting harder to find? Or is there another way to think about what's happening with these empirical observations? First of all, all of their examples are narrow. They pick a particular thing, and then they look at a particular metric. GPUs don't show up there. All of a sudden you get this ability to parallelize, and that's really interesting. There are a lot of external consequences. Basically they have these simple quantitative measures. They look at it in agricultural productivity. They look at it in a whole lot of different ways, but you do have to focus narrowly. I'm certainly interested in the fact that new types of progress keep becoming possible. But I think even there, there does still seem to be some phenomenon of diminishing returns.
Causes of Diminishing Returns
Is that intrinsic? Is that something about the structure of the world? What is it? One thing which hasn't changed that much is the individual minds which are doing this kind of work. Maybe those should be improved as well, or some feedback process going on there. Maybe that changes the nature of things.
I look at scientific progress up until, let's say, 1700, and it was very slow, and also very irregular. You had the Ionians back five centuries before Christ doing these quite remarkable things, and so much knowledge would get lost, and then it would be rediscovered, and then it would be lost again. You'd have to say that progress was very slow. It's partially just bound up with the fact that there were some very good ideas that we just didn't have. Even once you've had the ideas, you need to build institutions around them. You actually need to solve a whole lot of different problems about training, allocation of capital, and all these kinds of things. Even just basic security for researchers, so they're not worried about the Inquisition or things like that. There are all these complicated problems.
You solve all those complicated problems, and then all of a sudden, boom, there's a massive burst of scientific progress. If there's some kind of stagnation, if you're not changing those external circumstances, yes, you may start to get diminishing returns again. But that doesn't mean there's anything intrinsic about the situation. Maybe something external needs to change again.
AI and Instrumentation as Robots
Obviously, a lot of people think AI is potentially going to be a driver. It certainly will at some level. To that extent, you can think of a lot of modern scientific instrumentation as really, at some level, robots. What is the James Webb Space Telescope? It's unconventional maybe to describe it as a robot, but it's not completely unreasonable either. It is an example of a highly automated, very sophisticated system with electronically mediated sensors and actuators, where machine learning is being used to process the data. In that sense, we're already starting to see that transition. We've been seeing it for decades.
I have this "smoke a joint and take a puff" thought, which— I think we've had a few. I think we're getting to that part of the conversation, and then you can help me get my foot out of my mouth and figure out a more concrete way to think about it. To your point that there was the Industrial Revolution, the Enlightenment, and now there's AI, and each might be a different pace or a different way in which science happens. If you think about the pace of how fast such transitions have been happening, you can draw over the long span of human history this hyperbolic rate of growth that is increasing over time as well.
A hundred thousand years ago, you had the Stone Age. You go back even much further, how long have primates been around? It would be millions of years. A hundred thousand years ago, the Stone Age, then ten thousand years ago, the Agricultural Revolution, then three hundred years ago, the Industrial Revolution, each marked by this increase in the rate of exponential growth. Then people think it's going to happen again with AI. But that would happen potentially even faster. It would not have occurred to somebody at the beginning of the Industrial Revolution that the next demarcation in this trend will be artificial intelligence.
So if things are getting faster, and it's hard to anticipate what the next transition will be. I guess we just think of this singularity between now and AI as what distinguishes the past from the future. But applying the same heuristic that many people in the past should have had, maybe the "Intelligence Age" is also quite short and the next thing after that, we don't even have the ontology to describe what it is, the future will not think of the past as pre-intelligent AI and post-AI.
No, obviously we can't prove this, but it certainly seems quite plausible. Part of the issue is just that the substrate we have available to conceive seems all wrong. You can't speculate with a bunch of chimpanzees about what it would be to have language. Just to pick a major transition in the past, the transition itself is the thing. It seems likely.
Quantum Transitions and Speculation
If we're talking about "taking a puff" kind of thoughts, I'm certainly amused by the idea that there's going to be some transition involving artificial general intelligence using classical computers. But actually, there'll be an interesting transition with quantum computers as well. They're probably capable of a strictly larger class of potentially interesting computations. So maybe the character of AQGI, or whatever it should be called, is actually qualitatively different. So maybe there's a brief period between those two things. As I say, this is just speculation, but it's certainly amusing.
Is there a reason to think that? From what I understand, for decades people like you have put pretty tight bounds on the kinds of things quantum computers are going to do. It'll speed up search somewhat.
Shor's algorithm breaks encryption. From current perspective, it seems narrow, but we've only thought about it for 40 years, not deeply as a civilization. It could be narrow or very broad, a radical expansion. We've developed it without actual quantum devices, a major bottleneck. Analogy: computer science in 1700s with AND/OR gates couldn't anticipate Bitcoin or deep learning.
Inside view from 1990s-2000s contributor: Key transition making it a real field was around 1980. Computation became salient late 1970s-early 1980s: people could buy PCs like Apple II, Commodore 64. Quantum side: Paul trap enabled trapping single ions, manipulating single quantum states. Von Neumann pioneered computation and wrote on quantum mechanics but didn't invent quantum computing. Feynman and Deutsch papers in 1980s (Feynman 1982, Deutsch 1985) foundational, more comprehensive than earlier partial anticipations.
Feynman got one of first PCs around 1980-1981, excited he tripped and hurt himself carrying it. Contingent: talented quantum expert excited by new machines.
How Nielsen decided to pursue it: In 1992, took quantum mechanics class by Gerard Milburn. After fifth lecture, asked for papers. Milburn gave stack including Deutsch 1985, Feynman 1982, other fundamentals when few worked on it. Milburn wrote first paper proposing practical quantum computing approach in real system.
Papers exciting, fundamental questions, potential for progress. Deutsch's thesis: universal quantum Turing machine efficiently simulates any physical system (provocative, questions on quantum field theory). Ideas on quantum algorithms, wave function meaning—still debated. Sense of deep importance civilization lacks.
Market for follow-ups analogy: Like recognizing Shannon's theorems deep beyond pulse-code modulation, applying to quantum mechanics, genetics, computer science. Low-hanging fruit: provocative ideas to understand, tractably progress. Chose relative to interests/abilities where to dig. Unusual choice in 1992.
Did it work? Success: made open access papers, open code, open data salient. Meme: publicly funded science should be open science. Engages political economy of science.
Historical parallel: 3 centuries ago, Galileo/Kepler disclosed oddly (anagrams to claim priority). Took over century for modern ideals: disclose in papers, attribution, reputation economy for careers. Printing press/journals enabled it.
Now: share code/data/in-progress ideas, but no direct credit. Socially constructed.
Preprint example: Physics had preprint culture (arXiv), biology didn't (changing now). Biologists: biology too competitive, protect priority. Physicists: physics too competitive, upload fast for priority. Attribution economy constructed by agreement. Changing it alters knowledge system.
Collective science: Groups progress where no individual understands all levels. Best example outside math: LHC. Immensely complicated. Accelerator physics conference: experts on inverse methods for particle showers/cascades (retroactively trace origins for signatures). Deep simulation of tracks. One person can't master: quantum field theory, detector physics, vacuum physics, data processing for Higgs boson. Papers with >1000 authors; high-level talk, not deep specialties.
Darwin gestated decades, wrote enormous letters (prolific in routine). Einstein 1905 extraordinary: special relativity, photoelectric effect (Nobel), still multi-Nobel year without them.
Two work types:
- Routine: avoid procrastination, get good/outsource, do rapidly.
- High-variance: take time, go places/talk people (most not input).
Balance hard; personality trait preference. Einstein smart + luck.
Practice: identify routine, do quickly; bet on variance. In San Francisco: took 30-min beautiful walk vs 15-min efficient to remind of non-efficiency benefits.
Dean Keith Simonton equal odds rule: Probability any given output (paper/book) extremely important roughly equal; productivity determines era's output. Successful scientists publish a lot (Shakespeare). Counterexamples: Gödel published little. Avoid obsessively waiting for 'great project'—often fear public judgment.
Wish for biographies of talented failures (e.g., IMO gold medalists who failed as mathematicians)—more informative.
Label beyond 'writer': Darwin a writer?
Podcasting learning challenge: Ensure compounding across guests vs superficial depreciating knowledge. Many podcasters interview more experts, not wiser.
Advice for deeper learning: Make higher-growth/demanding context.
- Small: compounding returns.
- Radical: parallel different project. Demanding changes response.
With students/self: if $1M at stake, more effort? They've 'tried' but not really.
Prep examples:
- Chip design: brainstorm 5 roofline analyses.
- Ilya Sutskever: implement transformer.
Problem: no curriculum/exercises for most fields. Good podcasting possible without deep understanding—want to internalize/integration.
Fix: Change work output structure. Reject constant flow myth. High performers flow sometimes (athletes in game), not training (stuck/badly).
Choose guests with legible curriculum. No prep for Terence Tao.
Failure modes: Local maximum, not deep enough per topic. Learning needs deeper mechanisms vs black-box inputs/outputs (fades).
AI easy: implement. History? Ada Palmer interesting, learned some; post-interview 2,000 words consolidating/connections valuable. Spread episodes, more post-time.
Ideal: someone for curriculum/practice problems/exercises post-interview to clamp learning.
It's hard to find someone who could cover every single discipline. Maybe hire different experts for different topics. The key is identifying the problem you're solving for each episode. Understanding comes from getting interested in something, sensing a contribution to make, and honing in on a problem.
Spending time stuck is incredibly important. That used to just be annoying. Now it seems like it's maybe even the most important part of the whole process. That hard-won nature of it means that I internalize it afterwards.
I've written 10,000-word essays in a couple of days or in three months or six months. Essays that took only a couple of days taught little. Essays that took three months are still remembered 15 years later.
Learning involves a creative artifact: a class, engagement with a group working on a collective creative artifact (even as an input to their ends), an essay, or a book. Podcasts are enjoyable for this reason. Agreed to this interview partly because the questions are unusually demanding, providing a different forcing function. To pick the most demanding creative context.
For this interview, went through three lectures of the Susskind special relativity book. It has almost no practice problems, so hired a physicist friend to create a bunch of practice problems for every lecture. Planning to be appropriately humbled.
To make it as jugular as possible, raise the stakes as high as possible. An interview is high stakes but doesn't necessarily test deep understanding—it's not writing a book about special relativity or one that replaces the existing standard textbook. That's really high stakes.
People will talk about "going deep" on a subject, and it turns out different people have different ideas of what this means. For some people it means they read a couple of blog posts. For some people it means they read a book about it. For some people it means they wrote a book about it. The standard you hold yourself to determines a lot about your ability to integrate knowledge in this way.
Able to move faster on some things with AI, but unsure if learning better. The hardest, most demanding part is aversive, so people take any excuse to avoid it. Back-and-forth with an LLM lets you gloss over things—entertaining but not necessarily substantive. Easier because there's always a next question instead of intermediate thinking. Valuable to some extent, but seductive as a substitute for the real work.
Outsourcing judgment call: outsource routine low-value work to a chatbot.
Alan Kay, interviewed years ago on Linux: "It doesn't have anything to do with computer science. It's just a great big ball of mud. There are a few interesting ideas in there which are worth understanding, but mostly all you're learning is stuff about Linux. You're not actually learning anything which is transferable."
Seductiveness of Rube Goldberg machine-like systems: learning all the bits feels entertaining, but may not meet objectives. If goal is becoming a sysadmin, learning Linux is great. If understanding fundamentals of computing, much less clear.
Certain minds are seduced by learning systems and confuse it with understanding.
Will keep updated on revamped learning system within a month via text. Tiny incremental improvements in learning are worth so much—main input into the podcast. Bookshelves and blackboard are secondary; better learning makes the podcast better. Worth every morsel of improvement.
Thanks for the therapy session. Great note to end on.
Keep Dwarkesh Patel in your library
Save the videos and channels worth coming back to, and find them again in one place.
