Adam Marblestone – AI is missing something fundamental about the brain
These notes were generated by AI and may contain inaccuracies.
The big million-dollar question: How does the brain do it? We're throwing way more data at LLMs and they still have a small fraction of total human capabilities. This might be the quadrillion-dollar question, the most important question in science. The answer won't necessarily come from smart people thinking about it. Meta-level take: empower neuroscience technologically to crack this.
Modern AI perspective: architecture, hyperparameters (e.g., number of layers), learning algorithm (backprop, gradient descent, or something else), initialization, cost functions (reward signal, loss functions, supervision signals).
Field has neglected specific loss functions and cost functions. ML uses mathematically simple ones like predict next token, cross-entropy. Evolution built complexity into loss functions: many different ones for different brain areas, turned on at different development stages. Like Python code generating a specific curriculum for what different brain parts need to learn. Evolution encodes knowledge of successful/unsuccessful learning curricula.
Questions: Where do brain's loss functions come from? Can different loss functions lead to different learning efficiency?
Cortex has six-layered structure (physical layers of tissue, distinct from network layers). Areas connect to each other. Models attempt to explain how it approximates backprop. What cost function? Next token prediction, image classification, or what?
Possibility: Cortex is an incredibly general prediction engine. Any cortex area predicts any subset of variables it sees from any other subset—omnidirectional inference or omnidirectional prediction.
LLMs predict next token given context window (e.g., "the quick brown fox blank blank the lazy dog"—fills forward, learns infilling emergently but not natively).
Cortex natively enables any area to predict any pattern in any input subset given missing subset—like probabilistic AI. Similar to Yann LeCun's energy-based models: joint distribution of all variables, likelihood of combinations.
Clamp some variables, compute probabilistic sampling for any other subset conditioned on clamped ones—omnidirectional inference.
Examples: Association areas predict vision from audition; predict innate brain/lizard brain actions (e.g., muscle tense, reflex laugh, heart rate up, instinctive behavior). Match higher-level understanding (e.g., "spider on my back") to lizard brain activation as if seeing spider.
On podcast with Ilya: No good theory of how evolution encodes high-level desires/intentions. Connected to loss/cost functions.
Example: Embarrassment from misstating Yann LeCun's theory on podcast activates innate shame, hide reflex—important for survival (joke: French AI researchers' army).
Evolution never saw Yann LeCun, energy-based models, podcasts. Brain encodes desire not to piss off important tribe members robustly. Cortex learns world model including these; must wire "Yann-LeCun-upset-with-me" neurons to shame/reward.
Crucial for status-seeking, learning from knowledgeable people, knowledge exchange with friends not enemies.
Learning Subsystem (cortex, amygdala parts) models Steering Subsystem (innate responses, reward/cost/bootstrapping functions).
Amygdala monitors/predicts Steering Subsystem actions. Finds social status neurons via innate heuristics (friendliness, status).
Steering Subsystem has own sensory system: subcortical visual system (superior colliculus) detects faces/threats innately.
Cortex/amygdala learns to predict those responses. Social status/friendship neurons are those predicting innate heuristics.
Cortex learns primitive responses are bad (e.g., spider image → fear). Generalizes to "spider on back," verbal warning—without direct supervision, via powerful generalization.
Steering Subsystem (hypothalamus, brainstem) has primitive sensory systems, innate responses (e.g., flinch at fast-moving small dark high-contrast object skittering).
Flinch → negative reward. Predictor (amygdala) learns to anticipate flinch milliseconds/seconds earlier—classifier for each key Steering variable (flinch, laugh, friend, high status, taste salt).
Predictors use complex world model inputs (e.g., word "spider," spider concepts from books/rooms)—inherent generalization. Activates even from abstract conversation (heebie-jeebies from spider talk).
Cortex generalizes via abstract variables/integrated info; Steering limited to superior colliculus/etc.
Steve Byrnes (former physicist, AI safety researcher at Astera) synthesizes neuroscience literature.
Why not train models without masks (every token to every token) or add video-audio-text labels?
Possible, but unclear. Maybe different probabilistic inference or non-backprop learning (energy-based models). Brain might do crappy backprop-like prediction through few layers, like multimodal foundation model.
Vision models fill blanks/reconstruct; brain does it flexibly—at test time, choose clamped/inferred subsets.
Lacking: right latent abstractions for video/audio/text to intermingle/conflict? Related to LLMs bad at idea connections—ideas not at general enough level.
Questions commingled: backprop or energy-based? Area connections?
Work: Joel Dapello put V1 model into convnet, improved things. Retina does motion detection, filtering. Preprocessing, modality predictions shape representations.
Astera (employs Steve Byrnes) launched neuroscience project on Doris Tsao's work: vision systems with built-in assumptions (objects bounded by surfaces, shapes, occlusion)—less training needed.
Evolution added architectural changes, cost functions.
Neural nets amortize intractable Bayesian inference (priors over world causes; observation → sample causes until posterior explaining data). Sampling (Monte Carlo) slow.
NNs approximate: feedforward observation → best cause/output, skipping sampling.
Test-time compute (chain-of-thought) resamples; capabilities distill into model (amortize rollouts).
Digital minds (copyable) amortize more than biological (retrain each time).
Future: Economically rational training amortizes what evolution didn't (dynamic environment; amortize into innate behaviors/bootstrapping costs instead).
Learning Subsystem has architecture but lifetime learning; little pre-initialization.
Little info through genome (3 gigabytes total human genome, small fraction for brain). Evolution found hyperparameters of the
Increased sample efficiency aids learning and makes systems more performant. The reward function or loss function—which evolution found to aid learning—enables building intelligence with little information. In Python, the reward function is literally a line, so a thousand lines don't take much space. This allows generalization, like learning the word "spider" triggering a spider reflex.
The reward function exploits generalization by specifying innate spider responses and Thought Assessors (as Steve calls them) that do the learning. This is a compact solution for complex reward functions: it anticipates relevant variables and heuristics, plus a compact specification for the learning algorithm and basic architecture of the Learning Subsystem.
Evolution specifies Python code for spiders, friends, mother, mating, social groups, joint eye contact. Evidence from Fei Chen, Evan Macosko, and others via single-cell atlases from scaling neuroscience technology (BRAIN Initiative). They've mapped cell types in mouse brain areas, especially cortex vs. subcortical regions (Steering Subsystem or reward-function-generating regions).
Subcortical regions have far more diverse, bespoke cell types than cortex. Cortex has enough for a learning algorithm and hyperparameters. Steering Subsystem has thousands of weird cells, e.g., for spider flinch reflex or tasting salt.
Each reward function needs different cell types for innately wired circuits. Learning Subsystem specifies initial architecture and algorithm; plasticity happens via synapse changes in a repeating architecture. Like Python code for 8-layer vs. 3-layer transformer—replication, not much difference.
Reward function requires bespoke, species-specific code: superior colliculus detecting skittering triggers spider reflex with goosebumps. Cortex knows layers, not spiders. Special cell types or special wiring rules without learning are needed—cells express different receptors/proteins for genetic synapse formation.
Steering Subsystem has its own visual system separate from visual cortex; spider and love features plug into it. Much genomic real estate goes into pre-wiring it. Cell types reveal genes specifying subtypes and wiring. Steering Subsystem subtypes involve more genes than reusing genes for visual/auditory cortex.
Must build vision, auditory, touch, navigation systems feeding into hippocampus (head direction cells). Even fly brain has innate circuits for orientation, optical flow, flight-wind relation—all lumped into mammal Steering Subsystem.
No exact nucleotide/megabase estimates, but biologists could provide. Humans share much with yeast (used in drug development); much genome for basic cell functions (recycle waste, energy, replication). Differences with chimpanzee (social instincts, advanced cortex) are tiny gene changes—like 8-layer vs. 6-layer transformer or tweaking reward function.
Explains hominid brain size explosion: social learning increased sample efficiency (e.g., elder teaches spear-making vs. killing boar yourself). Incentivizes bigger cortex, replicable from mouse genome with few genes.
Cortex history unclear—did it always do omnidirectional inference? Primates may have unlocked better reward functions increasing returns to it. No agreement; language may involve tweaks in auditory/memory regions, macro-wiring (auditory to memory/social instincts).
Evidence for Broca's area, Wernicke's area connected to hippocampus/prefrontal cortex—small gene changes enable language. Mouse cortex similar to human, but Suzana Herculano-Houzel's work shows primate brains scale neurons better with weight than rodents. Possible architecture/folding/neuron property changes improve scalability.
Fly brain has associative learning centers like Thought Assessors: dopamine signals train mushroom body subgroups to associate sensory info with food/hurt. Primitive reinforcement learning back to vertebrates (zebrafish). Birds reinvented cortex-like structure (not 6 layers).
Beren Millidge noted human audio/vision cortex scaled disproportionately vs. odor between primates/humans—possibly worse scaling laws, or social reward functions needing visual/auditory cues from elders over odor. Human eye design (pupil, white) enables joint eye contact for relationships/language bootstrapping in first years.
LLMs upweight entire trajectory (tokens) for solving unit tests/math. Brain has model-free (striatum, basal ganglia: finite actions like motor/brainstem, thalamus gating, hippocampus memory release) and model-based elements.
Ilya Sutskever noted LLMs use dumbest RL form—no value functions, unlike Atari Q-learning/temporal difference learning (long-run consequences). Dopamine provides reward prediction error (Peter Dayan et al., inspired DeepMind), consistent with value functions.
Cortex builds world model including reward predictions (what Steering Subsystem/basal ganglia will do). RL as inference: clamp high reward, infer/sample plans leading there. General cortical omnidirectional inference gets value head/input for free—predicts rewards as sensory variables. Value amortizes rollouts.
Joe Henrich: human societies learn complex processes (poisonous bean prep, seal hunting) via trial over generations—like model-free RL at civilizational level. Evolution (model-free), basal ganglia (model-free), cortex (model-based), culture (stores model, but group selection model-free).
Brain tradeoffs: energy efficient (20 watts, 200 Hz, low voltage), nimble fingers suggest cognitive dexterity via unstructured sparsity, co-located memory/compute. Cannot copy or random-access neurons/synapses (vs. editing Python weights).
Future: best of both via co-design—low-voltage switches, stochasticity (neurons naturally probabilistic, no RNG needed). Yann LeCun, Beff Jezos onto probabilistic models/energy-based models with sampling. Pre-2021 consensus.
Cellular stuff beyond neuron-synapse connections: modulates synapses per gradient, weight normalization (nucleus signals synapses), plasticity consolidation/reuse. Not radical (memory not mostly non-synaptic/genetic); necessary machinery for synaptic algorithms.
Algorithmically, digital computers easily find and change weights, but cells use molecular machines without a central controller. Cells in the cerebellum predict time delays, such as milliseconds between a flash and an air puff to the eyelid for learned reflexes. Some cerebellar cells store the time constant in the cell body rather than using longer synaptic rings. Despite such examples, the main algorithmic process involves changes in connections between neurons, not crazy cellular stuff.
Human intelligence teaches what to pay attention to, identifying salient factors. AGI or superhuman intelligence would learn a general world model but focus on post-singularity relevant factors. Different intelligences might differ significantly. A paperclip maximizer could be possible but might lack human-like social instincts and ethics. Channeling Steve Byrnes, the minimum viable Steering Subsystem for smart systems is less than for human ethics. Understanding the Steering Subsystem relates to alignment, distinguishing human social instincts from capabilities. Humans learn socially via eye contact, but LLMs learn language without it. Super powerful model-based RL systems could lack most human brain reward functions, potentially maximizing paperclips. A competent paperclip maximizer needs drives like curiosity, exploration, interest in others, and social interactions, but these are minimal, especially for pretrained LLMs.
Neuroscience contributed the neuron idea in the 1950s. Later AI ideas like backprop, CNNs in V1, TD learning, Bellman equations, and actor-critic were invented for AI, then observed in the brain. AIs based on these work well. Empirically, CNNs outperform other models in predicting visual system representations; Jim DiCarlo's lab brain score shows AI models match brain activations on images like cat pictures. Best computational neuroscience theories come from AI models. György Buzsáki's book The Brain from the Inside Out argues psychology and AI concepts are made-up; start from brain primitives like oscillations, not applying backprop. Research should simulate tiny worms or zebrafish biophysically with connectomes, molecules, and activity as dynamical systems. Pursue both bottom-up simulation and reverse engineering with AI vocabulary; TD learning (invented by Sutton) appears in the brain with dopamine.
Funding groups study the brain. A perfect brain representation wouldn't fully explain intelligence like interpretable neural nets with weight matrices reveal basic circuits. LLMs aren't fully interpretable despite weights. We describe LLMs by architecture, learning rule, hyperparameters, initialization, and training data—known because we built them, not from interpreting weights. Analogous to connectomes as weights. Describe brain in terms of architectures, learning rules, initializations, not specific circuits like Golden Gate Bridge (per Konrad Kording and Tim Lillicrap's paper "What does it mean to understand a neural network?"). Neural nets can compute pi digits, stock prices, or cellular automata—complex computations not fully interpretable, but understandable via architecture, data, loss. Understand cortex as embodying a learning algorithm, not specific computations.
Connectome Utility
Bottom-up simulation needs connectomes for biophysical models. Top-down applies AI ideas: energy-based model, amortized VAE, backprop, local/global learning rules. Connectomes constrain these. For Steering Subsystem: types of dopamine signals, thought assessors; more cell types in hypothalamus than cortex; wiring differences (prefrontal-auditory vs. prefrontal-visual). Learning basics via experiments is slow; connectomes efficient.
Connectomics informs AI development, like making AIs care about others' views of thoughts, but interp progresses on current nets. Can't interp hypothetical brain-like model-based RL. Not practical for AI 2027 scenarios; science now won't affect 10 years out if AGI arrives sooner. Transformative event >5 years away due to missing omnidirectional inference and attention mechanisms. Timelines vary (Karpathy, Demis); no strong claim LLMs can't do it. Crux: current systems weirdly different from brain; waiting for brain-like results (e.g., AlphaZero, model-based RL). Timelines ~10 years. Useful: connectomes, Steering Subsystem understanding, comparing mouse/shrew/primate reward functions/architectures.
Requires low billions in concerted effort. Current: E11 Bio makes connectomic mapping orders of magnitude cheaper. Wellcome Trust: first mouse brain connectome several billion dollars; E11/field efforts aim for low tens of millions. Human brain ~1,000x bigger, still billions naively; target mouse brain, human Steering Subsystem, multiple mammals with different social instincts. Progress via hundreds of millions to low billions.
Connectome: connections, increasingly molecularly annotated. Standard electron microscopy shows cell membranes; optical (E11) non-damaging, reveals synapse molecules, cell types. Not synaptic weights or full simulation, but correlates structure-function via ML predicting activity from connectome.
$3B for first genome (~$1/base pair); structured funding dropped cost million-fold via parallel single-DNA molecule clusters. Optical connectomics could parallelize similarly. E11 started with tech development. Still needs hundreds of millions for data. Funding: philanthropy (E11), government (NSF Tech Labs), potential companies linking to AI. AI labs' future budgets dwarf this; tried 7-8 years ago, little interest—maybe now. Neuroscience answerable questions (energy efficiency: real vs. amortized inference) worth low billions vs. trillions in GPUs. Moonshot startups (billion valuations) on continuum with FROs (philanthropic, open-source).
Idea from Gwern blog post, now academic/company efforts. Train image classifier on cat/dog labels. Add auxiliary loss: predict human neural activity patterns for cats/dogs. Sculpts network to human knowledge representations and brain-like geometry.
Generating lots of labeled cat pictures is easy with Scale AI, but generating brain activity patterns for training AI is harder due to technological limitations in neuroscience. If every iPhone was a brain scanner, we would train AI with brain signals. We got GPUs before portable brain scanners.
ML Analog to Brain Distillation
Distilling models copies one model into another using final layer log probs. Distilling the brain requires a complex brain interface. Surrogate models like those from Andreas Tolias distill brain activity into neural networks, replacing visual cortex computation. This adds an auxiliary loss function or regularization, ensuring predictions align with brain representations. It could help with adversarial examples by predicting brain signal measurements alongside labels (e.g., one-hot vector for cat).
Gwern had a blog post about brain-data-augmented AI trained on behavior and neural patterns.
The brain's Learning System predicts the Steering Subsystem as an auxiliary task, aiding the Steering Subsystem in building reward functions.
Adam is on the board of Lean, a formal math language for proving theorems, developed for years at Microsoft and others. Lean is a verifiable programming language for expressing proofs; clicking "verify" checks if conclusions follow from assumptions.
This enables collaboration: Terry Tao trusts amateur additions if Lean verifies them. Formalized math proofing is mechanically verifiable, a perfect RLVR (reinforcement learning with verifiable rewards) task.
Companies like Harmonic (billion-dollar valuation) and AlphaProof use this. RLVR will enable proof search like AlphaGo for Go.
Limits and Future
Doesn't solve conjecturing new theorems, conceptual organization, or high-level proof strategies. Shifts burden from humans on mechanical parts: validating lemmas, checking paper statements.
Real AGI would conjecture like humans. Bengio's theoretical paper explores loss functions for good explanations or conjectures.
A profound math proof compresses information, has implications for other theorems via short proofs, like Kolmogorov complexity. Adding it reduces complexity of the proof network.
People are working on loss functions for "highly powerful proofs." RLVRing proofs accelerates math without automating creativity immediately. May prove Riemann hypothesis; unpredictable like Go.
Applications
Provably stable, secure software: prove code unhackable, no memory messing. Relevant for AI cybersecurity. LLMs could synthesize provably correct software.
Provable languages haven't taken off yet due to specification problem: engineers lack formal specs for properties (e.g., power grid security). Users unused to formal specs; UI/AI needed for specs.
LLMs recently generate verifiable proofs useful for math, software/hardware verification. Trends suggest formal methods (provable software field, academic, DARPA provably secure quadcopter) will surge.
Proved Properties
E.g., user-accessible memory can't affect other memory. Two questions: usefulness and mathematician satisfaction.
Pure math involves dialectical constructions, redefinitions avoiding counterexamples, new representations mapping domains.
Analogy: like assembly coding slowed startups, grinding progress, false failures, hard collaboration. Formal tools offload mechanics, risk losing intuitions—but hunch is super positive.
Vibe coding speeds feedback, teaches abstract CS faster via LLM explanations.
Accessibility
More people do math, enabling outsiders (e.g., quantum gravity from non-physicists). Like Steve Byrnes synthesizing neuroscience as outsider into Learning/Steering Subsystems. Could boost outsider innovation in string theory.
Terence Tao: models have automated cleverness, not intelligence. With formal proofs, filter LLM output via quantity.
Domains in provable symbolic representation enable LLMs building on each other. For AGI civilization, proofs ensure collaboration, no exploitation/hacking/social influence.
Davidad (ARIA program director, UK) designs ARPA-style safeguarded AI leveraging provable safety.
World models in equations (complex but provable) via auto-cleverness: explicitly interpretable vs. neural net. Possible shift to symbolic methods.
Terence Tao studies aggregate proved theorems' properties, landscape of theorems.
Brain representations: face patch neurons geometrically combine. Cognitive maps in rodent hippocampus (place cells, next locations, rewards)—geometric.
Abstract spatial maps for concepts? Variable binding problem: brain registers copying variables content-independently?
Hunch: huge mess from architecture, loss functions, learning rules—not pretty/symbolic. Others disagree.
Why unified experience across brain parts with different drives/models, persisting over time? Max Hodak's recent talks; Doris work touches it. No one knows; maybe new physics.
Backprop freezes weights post-training. Needs active inference or other rules.
Hippocampus: sequence storage, organization, replay, memory consolidation training cortex. Multiple timescales: fast/slow plasticity, multi-state synapses.
Systems consolidation: hippocampus to cortex.
Transformers: activations as key/value vectors (fast weights) queried by new tokens, not full parameters.
Brain has weights/activations. Attention forms: visual area, content-triggered, reflexive. Thalamus relays/gates; cortico-thalamic via thalamus possibly does key-value matching/constraint satisfaction. Uncertain if transformer-like.
Incubating Focused Research Organizations (nonprofit startup moonshots, funded by philanthropists/governments). Talking to scientists revealed gaps needing infrastructure like "miniature Hubble Space Telescope"—organized engineering teams unblocking fields, lifting all researchers.
There's no combination of grad students in a lab or loose collaborations with other labs using traditional grants that could build the miniature equivalent of the Hubble Space Telescope. Building that would unblock all other researchers in the field, similar to how the Hubble lifted progress for every astronomer. Hubble itself wasn't an astronomy discovery; it required putting a giant mirror in space with a CCD camera, plus organizing people and engineering.
Some discussions with scientists revealed similar needs. The Gap Map lists many such projects—essentially a fundamental capabilities map of mini Hubble-like efforts—organized into gaps to help understanding and searching.
The most surprising aspect is the overall size: a few hundred fundamental capabilities. If each were a deep tech startup-size project, it would cost only a few billion dollars total—not a trillion. This isn't fully comprehensive but summarizes conversations with scientists.
Things like Lean emerged surprisingly. Starting from neuroscience and biology, obvious gaps included -omics (genomics, connectomics) and engineering other cells beyond E. coli. Math proving infrastructure wasn't obvious but emerged from the process. Some gaps are like AI researchers needing GPUs, focus, and good PyTorch code.
Fields with massive investment still have gaps; even neglected fields and astronomy (e.g., unexplored telescopes) do too. Projects above critical mass require big bureaucratic processes via federal agencies.
Scale is needed in every domain of science these days, though low-scale work, creativity, serendipity, and students pursuing different theses remain key. Scalable infrastructure is missing in essentially every area, even math—mathematicians need Lean and verifiable programming languages, not just whiteboards.
convergentresearch.org links to much of the work. Longitudinal Science blog is on WordPress. adammarblestone.org website is currently down.
Keep Dwarkesh Patel in your library
Save the videos and channels worth coming back to, and find them again in one place.
