Building AlphaGo from scratch – Eric Jang
In a Nutshell
Eric Jang rebuilt AlphaGo from scratch using modern compute, achieving strong Go bots for ~$10K via neural-enhanced MCTS—combining policy/value networks with PUCT-guided search for efficient policy improvement through self-play and distillation, far cheaper than DeepMind's original millions. MCTS elegantly sidesteps RL's high variance by iteratively relabeling states with search-improved soft targets, enabling stable supervised learning that hill-climbs win rates without exploration failures or long-horizon credit assignment issues plaguing naive policy gradients in LLMs. Key takeaways: initialize with expert/small-board data, prioritize bug-free baselines before scaling laws, and leverage LLMs for hyperparameter tuning/experiments while humans supply research taste for track selection.
These notes were generated by AI and may contain inaccuracies.
Building AlphaGo from Scratch – Eric Jang
Eric Jang was most recently vice president of AI at 1X Technologies, and before that, senior research scientist at what is now Google DeepMind Robotics. He's been on sabbatical for the last few months, rebuilding, improving, and hacking on AlphaGo. The discussion covers building AlphaGo from scratch and what it tells us about the future of AI research and development.
Why AlphaGo is interesting: AlphaGo got Eric into the field. Early breakthroughs in 2014, 2015, 2016 showed how smart AI systems could become and the computational complexity class they could tackle with deep learning. Go was long understood to be intractable for search, yet solved through deep learning, which was mysterious.
Eric's training is in deep neural nets for robotics, where decisions are intuitive. But AlphaGo involves very deep search, mysterious how a ten-layer network amortizes simulation of something deep in the game tree.
Plot of compute for strong Go bots: In 2020, open-source KataGo by David Wu from Jane Street achieved 40x reduction in compute to train a strong Go bot tabula rasa. Not certain if stronger than AlphaGo Zero, AlphaZero, or MuZero, but very strong; most Go practitioners train against it today.
Thanks to LLM coding, what took DeepMind team, millions of dollars, research and compute, now done for few thousand dollars of rented compute.
Go is simple game, easy to implement on computer. Objective: put down black and white stones to occupy as much territory as possible.
Black always goes first. To capture opponent's stones: surround all four neighbors of an intersection; stone cut off from oxygen, dead.
Slight variations: Chinese, Japanese, Tromp-Taylor rules. Tromp-Taylor designed completely unambiguous; all Go AIs train and resolve against it.
In typical Go, humans can't put stone in instant suicide position. In Tromp-Taylor, fine; immediately resolves to death, same outcome.
Example play: Random play to surround stones. Move exposes empty neighbor, akin to check in chess. Must respond or lose stone. Surrounded on three sides, threat of capture.
Putting stone forces response, otherwise opponent gets two-block. Responding elsewhere leads to capture of three stones.
In Go, okay to let opponent capture some stones if positions to capture more elsewhere. Lose battle but win war. As board size increases, micro vs macro dynamics more interesting.
In territory example, black surrounds area despite white claim.
Different rule sets score differently. Humans resolve by consensus: both say game done, agree on territories. If disagree, keep playing. Once value functions agree, Chinese rules resolve.
Tromp-Taylor unambiguous, algorithmic. Score: count stones controlled (unambiguous), then empty intersections not touched by opponent's stones.
Intersections connected to both colors count for neither. If configured certain way, white gets three points, odd because human knows white losing them. Big difference computer vs human scoring.
Game ends: player resigns or both pass consecutively.
Understand AlphaGo to implement it. Start with intuition of underlying search process, layer deep learning for efficiency.
Go: two players. Person (black, first) moves, AI (white) responds. Encode board: represent black (1), white (2), empty (0); at least three values.
AI chooses best move, unknown until game ends. No local reward; win only at end.
19x19 board: ~361 moves initially, decreases. Game 250-300 moves. Experts end early, but Tromp-Taylor requires full play.
Tree depth 300, branching ~361^300, more than atoms in universe (redundancies, symmetries reduce, but naive tree enormous).
Merging children: Different paths can lead to same state; child node shared ancestor. Branching decreases by one each time.
Computer scientists thought Go intractable this century; exhaustive search too large.
Go deterministic: compute best strategy to stay in winning futures.
AlphaGo breakthrough: neural nets make search tractable.
Don't build whole tree; interactively expand worthy leaves.
Early bandit algorithms like UCB1 inspired AlphaGo action selection (not exact for sequential Go).
UCB1: argmax_a Q(a) + exploration bonus.
Node data structure:
- Visit count N_sa: how often parent visited this node (states, infer actions from child since deterministic).
- Q_a: mean action value.
- P_a: probability of action from parent.
- Dictionary of children nodes.
LLMs vibe-code reasonable MCTS implementations.
AlphaGo uses PUCT (Predicted Upper Confidence with Trees): argmax_a Q(s,a) + constant * term.
Intuition: Q(s,a) average goodness (if knew tree, sufficient). But building interactively, explore vs exploit.
Exploration term rewards untried actions. Initialized zero; grows for unvisited relative to total visits n.
ln(n) grows slower than n_a, shifts from exploration to exploitation dominance.
UCB bounds regret.
Go deterministic, but pre-AlphaGo used Monte Carlo: random rollouts for expected Q under random distribution.
P_a: prior probability (naive: uniform 1/valid moves). Diffuse tree slow; importance sampling issue, few high-value paths.
At leaves (no moves, Tromp-Taylor end), assign value U (win/loss).
Backup: mean action value up tree. Q_b weighted average of children (depends on sampling).
Without nets, still intractable; many low-value actions.
Value function necessary: humans stop early (dozens/100 moves before), glance p(win) via implicit neural net amortizing playouts.
Neural net speeds search by estimating value without deep search.
Core idea: shrink tree breadth (policy net for good moves) and depth (value net truncate search).
Per turn: encode board, run MCTS (1000 simulations explore nodes), select best move by visit probabilities.
After move (human/AI), discard old tree, new root, search anew.
One thing kept for later.
One small addendum: you don't discard all of that. You keep one thing behind that we'll use later. Just like for Reiner, flashcards were made for this episode to help retain concepts. Ideally, an LLM could generate candidates to refine. To get high quality suggestions, a pipeline was designed where the AI ingests screenshots of the blackboard at right timestamps, makes SVG diagrams if visuals are helpful, runs the writing and drawing through a critic, and revises the card in response to feedback. It's very hard to accomplish this just by stacking LLM calls. This step-by-step recipe works much better with a durable agent engaging with the task across previous stages. The Cursor SDK was used to spin up an agent for each card. The Cursor harness saved work in designing custom context scaffold or tool calls for taking screenshots or making animations. These agents run in the cloud, so no need to leave a laptop open—just get an email with candidates to review. Check out the cards at flashcards.dwarkesh.com. Start building with the agents SDK at cursor.com/dwarkesh.
Now that we have a basic intuition of how moves are made with search, neural networks speed this up by providing an analog to human intuition.
There are two networks. The value network takes in a state and predicts win or lose—binary classification. The policy network induces a distribution over good actions. This is a one-dimensional flattened move distribution, but really a square grid—these are probability distributions over good actions. Both are categorical classification problems, trained like any classifier with deep learning, cross-entropy loss. Specific architecture does not matter much—tried a few; Transformers work, ResNets work. For small data regimes, ResNets outperform transformers and give more bang for the buck at lower budgets due to inductive bias of local convolutions. Transformers outperform residual convolutional networks when more global context is needed.
One interesting finding from the KataGo paper: useful to pool and aggregate global features throughout the network for global sense of connecting value from one side of the board to the other. On a large 19x19 Go board with battles here and there, CNN receptive fields compute local things and make them invariant, but won't connect distant features easily—they need pooling and attention. Vision transformers are good because of global attention, but need more data to learn invariant local features.
Tried hard to make transformers work for breakthrough in Go, removing tricks, but haven't figured out making them better than ResNets for now.
CNNs bias towards spatially nearby relevance, aggregated slowly. For games where local isn't relevant and whole board must be considered, transformers work better (spatial dimension). For temporal dimension in deterministic full-information games like Go, only previous move considered.
In poker or Diplomacy, past bluff relevant—need all previous states. In perfect information games like Go, Nash equilibrium strategy exists you can do no worse than any other. If opponent biased (e.g., aggressive), can counter better than Nash, but Nash decided from current state alone. AlphaGo design choice worked well—Nash superhuman, beats all human strategies.
Variations need temporal history: exciting research—fork repo and try. In 2v2 Go, model partner's behavior, aggregate info to respond. No longer perfect information. In imperfect/partial observability, need context to build model—exciting for self-play or Diplomacy-style games.
Architecture not super important—works with transformers or ResNets; ResNets better for low-budget. Use Karpathy-style AutoResearch hyperparameter tuning. Set up problem with target optimization. Arbitrary architecture that worked: encoded board state like RGB—three channels: black, white, empties (or masked for multiple board sizes—not covering now). Feed into ResNet, two branching heads: value head single logit R¹, policy head R³⁶¹.
Train to predict game outcomes from board state and good moves. Original AlphaGo Lee initialized with supervised learning from expert human play; later removed for self-teaching.
Super nice for implementation/audience: initialize experiments to something easy, get working before tabula rasa. In deep learning, initialization everything—initialize research to closest success, especially new things. Pick what works, improve; don't start from non-working.
Take human expert plays: predict actions where expert won; predict outcomes regardless of win/loss.
For early boards (one stone), with hundreds of thousands games, averages show half win/half lose—logit converges to 0.5. As game progresses (hundreds steps), win probability clarifies under expert data.
This value thinking true for any training data, not just expert—even tabula rasa. Vibe coding: gather expert datasets (KataGo online/human), train—model beats most humans. Argmax policy = fast, intuitive strong Go player (10 layers, <3M params)—miraculous.
Verify Go rules correct, simulations fast—checkpoint before layering search.
Can do better than raw NN—apply to improve MCTS.
Four-step iterative MCTS: selection, expansion, evaluation, backup. Pick simulations: 200-2048 (AlphaGo vs Lee: tens of thousands/move for max strength; training/KataGo: this order). Modern bots need less compute at test time—raw network takes TPU pod burden into one forward pass (TPU adds oomph).
Every simulation: select best moves in tree, add leaves, update action values.
Tree starts basic: root node (current board). Evaluate with NN for Vθ, policy probs. Create children (e.g., 3x3 board w/1 stone: 8 possibles, probs P₁ to P₈).
PUCT criterion: Q(s,a) + cPUCT × Pₐ × (√N / (1 + Nₐ))—initially N=Nₐ=0, biased to highest P. Select (e.g., P₁ highest), reach non-leaf/terminal—expand.
Run board through policy NN (AI's move; now opponent's). Expand leafs, evaluate Vθ per child (Node₁', etc.)—intuitive win guess for player. Zero-sum: value = 1 - opponent value. Vθ shortcuts full search.
Original AlphaGo Lee: average Vθ with real playout to Tromp-Taylor resolution (α Vθ + (1-α) playout). Playout: policy NN vs itself to end (not nested search)—grounds estimates, useful endgame. Unnecessary—dropped in later papers; speeds up (no rollouts/sim).
Q = average evaluated child values; running mean over simulations. Recurse up tree.
Iterate: select best, grow tree, NN guess win, backup to root. Visit counts reflect search-induced policy.
Final: visit counts vote for action.
Could normalize child V into policy? Fine, works—but policy prior from single forward pass prunes breadth efficiently (vs 361 passes/batch). More: MCTS recursively improves own predictions/search—explicit policy modeled better than implicit from value.
After simulations: sparse tree—high visits gravitate to best path (contrast exhaustive tic-tac-toe factorial tree).
MCTS improves value/policy networks.
Twenty years ago, Jane Street data center fit in office corner. Ron Minsky (co-leads tech) on The Hive: first version six Dell boxes stacked. Trading systems there to turn off easily—cleaner unplugged one mid-day vacuuming. Better in data center.
Data centers advanced—toured Texas one with Ron Minsky, Dan Pontecorvo (physical engineering lead). GB300 cabinets: 140 kW peak (vs air-cooled 10-40 kW). Liquid: deionized water + 25% propylene glycol (inhibits bacteria/algae). Full tour at JaneStreet.com/dwarkesh.
RL: AI gets stronger playing itself. Play game: opponent moves, AI searches—visit count distribution (initial policy). After MCTS, more confident on actions. Maybe the distribution looks a bit more
MCTS produces a peaky distribution based on search. Tune search to be more diffuse, but that's not ideal. MCTS should get more confident about specific actions. Initially places weight on other actions, but converges to peaky distribution as simulations increase. Call this π, wrapped in MCTS operator π(a|s). After MCTS, policy recommended distribution is more peaky. Take argmax or sample from it. Make move, throw away tree, begin anew.
Compute new distribution each move. Initial guess refined through MCTS. On every move, initial guess from policy network, then search combining policy and value network arrives at confident action.
Game ends, one wins, one loses. AlphaGo trains by taking final search process outcome and telling policy network to predict it directly from the start. This reduces MCTS legwork.
Test-time scaling plot: x-axis simulations, y-axis win rate. At zero sims, raw policy win rate. Increase sims, win rate curve rises. 1,000 sims gets to certain policy/win rate. Distill MCTS policy into shoot-from-the-hip policy network, start higher. With another 1,000 sims, reach even higher. Amortizes first 1,000 steps into policy, better result per sim.
Sigmoid nature of scaling: win rate gains diminish with more sims. Unsure if distilled network regains early gains or inherent to MCTS. Test-time scaling sensitive to policy strength. Monotonically increasing to 1.
Next iteration trains network to approximate 1,000 search steps outcome. Neural network starts stronger, play improves with additional sims. Training: apply search on every move in games (win or lose), train model to imitate search process.
Schematic: states S₀, S₁, S₂, S₃, suboptimal trajectory actions. Relabel to better actions A₀’, A₁’, A₂’, A₃’. MCTS gives strictly better action per step, even if game lost. Retrain policy on these tuples improves it. Related to DAgger in robotics/imitation learning: intervene even in bad states (e.g., self-driving car off road).
Pedantic: no guarantee MCTS > policy early (value network poor). Heuristic that works in practice. Example failure: good policy Πθ(a|s), but MCTS bad due to poor terminal values (e.g., bots resign before Tromp-Taylor resolution, forget late-stage evaluation). Bad leaf values propagate via PUCT, backups off, different distribution. Low sims cause variance, only converges as N → ∞. Inaccuracies screw policy recommendation.
AlphaGo Lee used playouts to end for grounding. Tip: 10% games prevent resign, resolve to end for late-stage data.
Assume accurate value functions, search gives better policy. Critical chain.
Cold-start/AlphaZero: first epochs, policy useless; play full games, label winner for preceding moves. AlphaZero loss: policy vs MCTS + value vs game winner, per move.
Trick: ensure good value function before MCTS (don't search on garbage). AlphaGo Lee: train on human games. Or open-source Go bot self-play. Offline realistic data learns late-stage value well.
Easy to evaluate late-stage Go: near-decidable, low uncertainty. Games to end train terminal value. Search backs up to intermediates. Value head learns healthy vs not-healthy boards (hardest mid-game). Analogous to TD learning (contrast Monte Carlo later).
Expert data shortcuts. For tabula rasa (AlphaZero/KataGo): random games on small board (9x9 sees patterns). Train 9x9+19x9x19, good transfer. 4x4 like Tic-Tac-Toe, random play gives human-like ends; 5x5/9x9 reasonable. Bootstraps policy improvement via search. Critical: ground value, else MCTS fails.
Sharing representations efficient? AlphaGo Lee: separate networks. Later: merged heads, saves compute intuitively (consistent predictions). Rigorous answer hard.
Predicting MCTS seems less impressive vs LLM RL (builds code from yes/no), but AlphaGo profound: 10-layer NN (10 reasoning steps, parallel/distributed) amortizes intractable search to high fidelity. Girds AlphaFold (compresses microsims). Questions P=NP understanding: NP-hard worst-case (Go? protein folding), but structure exploited. Reformulate NP-hard solutions?
Not exact optimum, but practical. AlphaTensor/AlphaFold too. Limit: complex sims (weather) compressible into NN forward pass. AlphaGo first showed simulation compression.
Chaos role? Weather hard (10x resources/day). Go/protein chaotic, sensitive. Analogy: board future sensitive (one stone disrupts). But predict macro (win prob, expectation over futures), not exact. Weather: hurricane position, not wind velocity. Lorenz attractor: micro chaotic, macro structured.
Hash: sensitive, no macrostructure. Cryptography unproven vs fast NN approx (relates P≠NP). RSA structure quantum exploits. Crypto/NN convergent: sequential jumbling. NN max power at edge of chaos (Jascha Sohl-Dickstein papers). Chaos not hopeless, useful at boundary.
MCTS doesn't directly upweight winning actions. Improves every action via exhaustive search supervision, low variance.
Naive: league agents, reinforce winner actions. Even match Πa, Πb (50% win). 100 games, 300 moves/game. Πa wins 51-49 (luck/perturbation). 50 equal games; 1 diff (critical smart move). 1 supervision amid 99×300 neutral moves. Sucking supervision through a straw (Karpathy). LLMs use policy gradient this way.
Scale to millions for signal, mask neutrals, use RL advantages/baselines. Gradient RL: high variance (sum rewards/return, win/loss indicator; no train on losses).
When you multiply these terms out to compute the variance of the gradient, it's equal to the expectation of the square of this product term. For simplicity, pretend this is on average zero if centering it at no signal. The variance grows quadratically with T due to a coupling effect on these terms.
In LLMs, the decoder predicts sequences like "hello world". Current LLM RL treats the entire sequence as a single action a_t, with T=1. Transformers formulate this through the product of conditional probabilities, so the log probability of the whole sequence equals the sum of individual token log probabilities: log(hel) + log(lo) + log(world).
However, when sampling with a reward per token, interaction effects arise from cross-multiplication of gradient terms and rewards. The challenge is ascribing credit from the episode to different terms, especially with reward only at the end.
Process supervision could give rewards per action/step, summing them at the end as one reward signal for the single log prob action. Decomposing into multi-step introduces correlations between actions, magnifying variance. With T=1, a single log prob and reward estimate still has variance from this term.
The naive REINFORCE estimator in LLMs resembles the return of a single action.
Penalize unhelpful labels and reward helpful ones, like training on neutral labels. Intuitively, find a training objective term that discourages neutral moves or nullifies their gradient effect while affecting good moves.
Optimal case: discard all neutral moves, gradient only on improved move. Advantage estimation subtracts a baseline from the multiplier, behaving like zero for neutral and one for better-than-baseline. For wins, distinguish truly better moves from accidental wins.
Use TD learning to approximate the Q-function. Subtract from return to push up above-average actions and down below-average (advantage). Recommend John Schulman's Generalized Advantage Estimation paper.
Requires good estimate of average state performance, linking to value function.
Model-free RL solves credit assignment without knowing good/bad actions. MCTS improves labels for actions rather than crediting wins.
Used in AlphaStar and OpenAI's Dota. Unifies model-free RL and self-play.
In games without easy tree search (e.g., StarCraft, partially observable, non-deterministic), train best response policy. Fix opponent Π_b (e.g., Zerg vs. Protoss), use model-free RL (TD-learning, PPO, etc.) to beat them. Reward: 1 for win, 0 otherwise.
Distill multiple best responses against league (Π_b, Π_c, Π_d) into a mixed strategy, averaging across opponents. Proxy for MCTS teacher: relabel states with better actions.
If win against policy, reinforce all trajectory actions using PPO, V-MPO, Q-learning.
MCTS: Tree with value function at leaves, back up mean value (Q from simulation averages).
Model-free: Q-values via TD learning or PPO advantage. Q-learning: Q(s,a) = r + γ max_a' Q(s',a'), recursive dynamic programming. Train networks to enforce consistency, propagating future Qs backward.
Recover policy via argmax Q. Q-learning (approximate dynamic programming) for value estimates without forward search; collect trajectories first.
Q-learning big for high-dimensional problems (robotics) without world models: collect trajectories, plan on reward.
LLMs do backward learning: reinforce trajectories passing unit tests (e.g., coding). Vs. MCTS forward per-move improvement.
MCTS sample-efficient: local search truncated by value function upgrades policy iteratively. LLMs learn per trajectory "through a straw", upgrading possibly irrelevant tokens.
Google 2023/2024 research applied tree structures to reasoning; jury out.
MCTS simple for Go: concrete value estimation, determined breadth/depth, PUCT suited to Go.
PUCT (√N / (1+N_a)) assumes resampling children; LLMs rarely resample due to vast discrete tokens. Language too broad/open-ended for discrete action heuristics.
In Go, MCTS better than current policy even without full trajectories. LLMs/robotics lack local evaluation independent of solving the problem.
Ongoing research on MCTS/MuZero for continuous control. High-dimensional/combinatorial spaces (language) resist Go-style heuristics.
Parallel future tracks could help; math more tree-like than business negotiation.
Andy Jones 2021 paper "Scaling Scaling Laws with Board Games": trade test-time compute (MCTS) for training compute.
Predicts LLM inference scaling. Forward pass learns sequential/recursive behavior.
Paper predicts LLM-style scaling laws and compute to solve larger boards (e.g., 3x3 to infinite Go).
Project motivation: build strong Go bot via Bitter Lesson/scaling laws, without KataGo tricks.
Not successful yet; scaling laws need working recipe/good datasets first. Get system working, collect data, then study scaling laws.
Initial mistake: scaling laws on buggy MCTS labeling/expert data (plotted curves, but bad data). Need good data/architecture/no bugs first.
Advice: don't study scaling on uninteresting artifact; fix bugs first.
Compute for best AI models: smooth exponential log line, except AlphaGo Zero outlier at 3E23 FLOPS (comparable to frontier LLM).
Speaker's bot: $10K donation from Prime Intellect; $4K exploratory, $3K final run.
First-to-do requires more compute than catching up (distillation, crutches). Speaker used best-response vs. KataGo; validating tabula rasa.
Research starts from good init; AlphaZero tabula rasa, prioritized working over efficiency.
Robotics frontier scatter plot messy vs. clean LLM line; not at scale where every FLOP counts. Optimize for capability emergence over efficiency.
Scaling forces efficiency (hundreds millions $). Past: plentiful/unaccounted compute → optimize time/speed/getting-it-working.
AlphaGo had lots of compute, no need for optimality.
No major open-source strong bot post-KataGo (2020). KataGo had clever ideas.
Bitter Lesson vibe: architecture (Transformer vs. ResNet) doesn't matter much; GPU speed makes model size irrelevant.
Simplify: synchronous collect/train SL/collect vs. distributed async RL (replay buffers, pushers, collectors).
Nvidia GPUs faster: KataGo on V100s, now half desktop Blackwell GPUs works.
Some of the auxiliary supervision objectives that KataGo developed aren't really necessary if you have a strong initialization. If you're initializing best-response training against KataGo itself, your own model needs none of the tricks that KataGo needs. The core thing is getting as quickly as possible to some strong opponents. That matters a lot more than the specific architectural innovations. But there are still some nice compute multipliers. Training on 9x9 boards was very nice for resolving endgame value functions. If you can co-train that on an architecture that can transfer between 9x9 and 19x19, you can really cut down the warm-start time to learn from scratch. AlphaGo Zero's plot showed the first 30 hours or so spent basically catching up to the supervised learning baseline. You can cut down that time a lot by pre-training on a small board and then warm-starting that into your 19x19 board play. There was some other stuff, like varying the number of sims between episodes. This turns out to be not that sensitive. You can fix it or increase it. It doesn't matter too much. It's nice from a scientific perspective, revisiting an old paper and seeing what really matters.
Why is it okay to have a buffer in AlphaGo? Every time I talk to an AI researcher, they're telling me about how bad it is to be off-policy. But the way a naive implementation of AlphaGo Zero would work is that most of the moves in a given backward step, or in a batch of backward steps, would not be among the ones made by the most recently trained model. So why is that okay? This gets into the fundamental off-policy versus on-policy reinforcement learning questions.
As you'll recall, in MCTS, you take actions that you took, and you relabel them to take different actions on the same states. The off-policy part comes in where, what if you're relabeling states that your new policy would never visit? What's the point? You're wasting capacity. In the extreme limit, imagine the distribution of states in your training buffer are all states you would never visit. Then you're supervising them to take good actions on states you would never achieve. Therefore, your policy can get really bad. This is where off-policy can really hurt AlphaGo.
However, if you interpret this from the DAgger perspective—which is basically a way to correct yourself back to the optimal trajectory given some data—what you want in an algorithm like this is to have mostly states that you would visit, but then a small or reasonable percentage of states in this high-dimensional tube around your optimal trajectories. Any of those states are given a supervision target to funnel you back into your optimal trajectory.
In a DAgger-style setup, your optimal training data distribution is your optimal states and actions. You want to be in this state, you want to be in this state, you want to be in this state, and then you win here. These are your optimal policy actions. These are the things you definitely want to train on. But to make it robust to disturbances, you want to make sure that if you happen to drift off into some other states, you can funnel yourself back in.
But why isn't this a fully general argument for off-policy training? This is actually why you want to do off-policy training sometimes. You don't want a compounding error where, if you make a mistake, you don't have the data for how to return to your optimal distribution. Optimal control doesn't really say too much about how to not accidentally get here, because it's making the assumption that once you learn the policy, you're going to get to here. But in applications like robotics, a gust of wind blows you slightly off and now you need to correct. Or the friction on one of your tires is slightly lower than the other wheel, and now your car is drifting and you have to correct it. These kinds of things often happen in more real environments.
The problem with Go and chess is that the other player is always trying to do some shit.
Things can drift off, and you always want to be able to correct back to your winning condition. Your replay buffer really should have the states your policy would visit, plus some distribution of states you might drift to, and then how to return back to your optimal states.
Now, if you take this to the extreme and you say, "We don't have any of this data, and we're just going to be labeling with MCTS states that are so far away from our optimal behavior, like this bag of states over here…" Well, now each of them gets an MCTS label, and your policy learns how to take the best possible action here, but you never get here. You're training your model on states you would never reach. This is where off-policy can really hurt.
As part of this project, I did try an experiment where I took a bunch of trajectories, and to try to saturate the GPU as much as possible, I took random states from the dataset and reran MCTS on just those states. Instead of playing a whole game where I'm doing MCTS on every move, I ignore the causality of moves and pick random board states, and I label those with my current network. I might revisit old states I've labeled before and relabel them again with my current network. In practice, this actually does work. You can take some states that are reasonable and constantly be relabeling them while training.
This starts to converge on a very robotics-like setup, which is very common. You have your dataset of trajectories, and then you have something like a replay buffer pusher. These are off-policy offline trajectories. Your replay buffer pusher pushes transition tuples to the replay buffer, and then you have a job that's continuously replanning what the best action you should have done instead of the action you took.
In robotics, it's very common to minimize TD error. Your Bellman updater is constantly pulling things from here and trying to satisfy Q(s,a). From here you have your trainer, which is trying to fit the Q to the Qᵗᵃʳᵍᵉᵗ. You can think about this as a sort of planner. You visit old states you've been to, and you take your current model and rethink. What could I have done better if I visited this? This is how off-policy robotic learning systems are usually trained. These days there's a simpler recipe, but in the Google QT-Opt days, we did things like this.
The trainer is where you try to minimize Q(s,a) and Qᵗᵃʳᵍᵉᵗ.
You have your off-policy data that came from various policies. You're constantly pushing transitions you saw before to a replay buffer. Then you've got this thing called a Bellman updater, which basically replans: instead of this action, what action should I have taken at s to have a better value? The way you enforce that is by trying to minimize the TD error. Given this, you have s’. You compute Q(s’,a) and you find the action that should go with s’ that makes this Q value as high as possible. Then you add that to the reward here, and that gives you your actual target. For this current s and a, your Qᵗᵃʳᵍᵉᵗ is this. Now you send back the Qᵗᵃʳᵍᵉᵗ to this transition. With this tuple, you pair a Qᵗᵃʳᵍᵉᵗ. Then on the trainer side, you simply use supervised learning and minimize your current network's Q(s, a) with its target. In the background you're basically thinking through how valuable all these actions actually were. In a more optimal policy where you're trying to maximize this, what is the Qᵗᵃʳᵍᵉᵗ of this transition? It's sort of like daydreaming. You can think about it like you're going back in hindsight. Given what I've seen in the historical buffer, was there a better action I could have taken?
The connection to Go here—I tried and it was moderately successful, but too complex to open source—was replacing this with an MCTS relabeler. Instead of doing this target network computation, you run MCTS on your transition. In this case, you have your state, your action, and whether you won or not at the game. Actually, you can just toss those two. You don’t care about these ones. You just take your state and plan MCTS to get your best policy, Π, on your current network. Not the network that took this action, but your current best policy network. You rerun your search offline on these transitions. If these are transitions your policy can get to, this actually acts as a very nice stabilizing effect.
Another benefit is that you can fully saturate your GPU better because you're not blocking on the Go game to give you board states. You simply search across all board states at any depth in parallel. Here the trainer would just predict the MCTS label as best as possible. This kind of works, and it is quite relevant in robotics where you have a lot of offline data and you can't simulate things like MCTS. But in practice, it does run into the problem that if the current model is looking at states it would never reach, it's wasting capacity. You have to be a little bit careful here.
Much of RL has converged to a much more on-policy setup where they don't really try to directly train on off-policy data. At best, they use off-policy data as a way to reduce variance, but not to directly influence the objective. Why have they converged to that? It's just more stable. You might use the off-policy Q as a way to do advantage computation, like Q minus the sum of Q. If there are N actions, this is your value, and these are your current Q values. Your advantage for that action is the average value minus your current one. People can try to estimate Q in an off-policy way and then just use advantage here. If there are a problem in these dynamics, it doesn't blow up your loss as much. In robotics, there's a convergence toward using off-policy data to shape your rewards but not actually be directly here.
I'm reminded of our earlier conversation about why MCTS is so favorable compared to the REINFORCE or policy gradient thing LLMs do. Policy gradient RL is even more inefficient than you might think. The inefficiency one thinks about naively is the fact that you have to roll out a whole trajectory to get any learning signal at all. As these trajectories become longer and longer—as an agent has to do two days' worth of work to figure out if it did a project correctly, rather than just completing the next word in a sentence—the amount of information per FLOP decreases. You have to unroll two days' worth of thinking to see if you implemented a feature correctly, so the amount of samples per FLOP has been decreasing. You're trying to maximize bits per FLOP. You can think of bits per FLOP as samples per FLOP times bits per sample.
What I mentioned a second ago is that the samples per FLOP go down as RL becomes more long-horizon. But this kind of naive RL is also terrible from a bits-per-sample perspective, at least compared to supervised learning. Early on in training, let's say you have a vocabulary size for an LLM that is 100K long, so there are 100K possible tokens that one could answer. You have a totally untrained model and a prompt like, "The sky is…" With supervised learning, the model would have some probability distribution over all the things it could say. There's a label that says the term here is "blue," and it would learn through cross-entropy loss exactly how far its distribution is from correctly saying "blue."
If you were doing this through RL, the model would try, "The sky is halycon." Nope, that's wrong. "The sky is told." Nope, that's wrong. This is a totally untrained model. You would have to do this on the order of 100K times just to stumble on "blue" and get some learning signal. In the supervised learning regime, you have your distribution of probabilities, you get told that it's "blue," and you figure out how far off you were. The amount you learn is a function of your pass rate. The further away you are from "blue," the more you learn to go toward "blue" using cross-entropy loss. You can think of it as like your pass rate, your prior probability of having said "blue."
As a function of that, in supervised learning, through cross-entropy loss, you would learn negative log(p), p being pass rate, bits, once you get this label. Whereas in RL, if you're just randomly guessing and seeing if it works, that's basically just the entropy of a binary random variable. What's also tough here is that the distribution you're sampling under is your policy's distribution. If your policy has no chance of sampling "blue," then you will never get a signal. That's modeled by the fact that your probability of sampling "blue" is extremely low. If you sample it, you do learn as much as you would have in supervised learning. In all other cases—99.99% of the time in an untrained model—you're learning incredibly little from seeing that "halycon" or "told" is not the correct word.
If you put your pass rate on the X-axis and the bits you're learning from a sample on the Y-axis, with 0%, 50%, and 100%, so at the end of training you're here. If you have supervised learning, the negative log pass rate would look something like this. The entropy of a binary random variable would look like this, depending on whether you're doing nats or bits. If you do bits it's one here at the peak. This is like a coin flip. You learn the most from a coin flip. This is supervised learning. This is RL. However, the problem is you spend most of training in this regime, in the low pass rate regime. How fast you're learning is a function of how many bits per sample you're getting, and you're getting very little signal here.
If you chart the pass rate on a log scale—where at the beginning of training with a vocab size of 100K the pass rate is 1/100,000, then 1/10,000, 1/1,000, 1/100—what this graph looks like here, supervised learning would look like this, and then RL would look like that. And arguably you spend all your time here, potentially never even getting a single success. It's a depressing plot in the sense that once you're here, it's not at all obvious how you get to there. Once you're here you have something, but you actually spend all the time here in many RL problems. There's a question of how you initialize so you're at least at a non-zero pass rate.
One more thing I'd like to add about bits per sample that's very relevant to any machine learning problem is that there's a connection to soft targets and distillation. If you have access to the logits, not just the one-hot token answer, if you have access to the soft targets the entropy of this distribution is far higher than the one-hot. There's way more information in bits per sample in a soft label. That's why distillation is so effective per sample. It's giving you way more information per sample. It would just be the entropy of this distribution. The entropy of this is zero. The entropy of this is the entropy equation.
This is also why AlphaGo is quite beautiful. In AlphaGo, you don't train the policy network to imitate the MCTS action. You train it to imitate the MCTS distribution. Both of these are valid, and if you wanted to do a scientific experiment of how important this soft label is, dark knowledge distillation, you can run an experiment where you retrain the policy network on the action MCTS selected rather than the soft target.
Intuitively, why is this ability to do iterative search—where you don't necessarily need to be able to win the game in the beginning, you just need to be able to improve your current policy—so powerful a capability in learning compared to how LLMs currently learn RL? It's exactly this thing of considering your pass rate of the entire trajectory.
Why is AlphaGo an elegant RL algorithm? The major reason is that you never have to initialize at a zero percent success rate and solve the exploration problem of how to get to a non-zero success rate. This is what allows you to hill-climb this beautiful supervised learning signal. If you look at the actual implementation of AlphaGo, every step of the way, there's actually no TD error learning or dynamic programming, at least explicitly. It's just supervised learning on a value classification as well as a policy KL minimization. It's just a supervised learning problem on improved labels. The training is very stable. You can train as big of a network as you want. You can retrain this on the data set. Everything will just go stably. The infrastructure is very simple to implement as well. You don't need a complex distributed system to keep everything on policy. At the end of the day, you're just saying, "I have some improved labels. Let's retrain my supervised model on these targets." You're always in this beautiful regime where you're just trying to improve the policy, rather than escape local minima where every signal is flat all around you.
One way to draw the curve is if you draw the win rate of an MCTS policy versus the raw network… Let's say this dotted line is the raw network. The MCTS policy looks like this. Every step of the way, this supervision signal is very clean. You're never in a situation where the MCTS is giving you no signal, unless your MCTS distribution converges to exactly what your policy network predicts.
You did a bunch of the research for this project through this automated LLM coding assistant loop. Observations about what the AI is good at, what it's not good at, what you think about this scenario's likelihood eventually.
Automated scientific research is one of the most exciting skills that the frontier labs are developing right now. It's important for everyone who's doing any kind of research to get a good intuition of what it can do now and what it can't, and how the science process might work in the future once we have AIs automating a lot of this investigation.
In brief, I mostly used Opus 4.6 and 4.7 while working on this. What works is that the models can do a very good job of hyperparameter optimization. In the past, people would come up with a search space of hyperparameters like learning rate, weight decay, and maybe how many layers are in your network. They would do a grid search or a Bayesian hyperparameter optimization approach, and it would find some tuned parameters.
The really cool thing that automated coding can do now is search a much more open-ended set of problems. It can say, "I've identified that the gradients are small in this layer, so let me change it up here. Let me rewrite the code so the data loader has a new augmentation I came up with. Let's try to find the best way to fit the constraints of the optimization problem." You end up with this much more flexible, high-level, almost grad-student-like ability to just grind a performance metric. This can squeeze out quite a lot of performance. On a fixed data set with a fixed time budget, you can improve perplexity by quite a lot on a classification problem like LLMs or Go.
It is also fantastic now at basically executing any experiment. I have a Claude Skill that I wrote called Experiment where I give it a description of what I want it to plot. I just describe, "Here's the x-axis I want, here's the y-axis. Answer this question for me." It'll run off and do all the experiments, compile the plot, make a report, and suggest what might have caused it and so forth. That's what works quite well today, and I think we can expect these abilities to get better in the future.
But it's also useful to know what it's not doing so well today. In the blog version of this tutorial, I have a plot of all the experiments I did grouped in a tree, where every node represents a failed, successful, or mixed experimental result. From there, it branches off into a child representing the follow-on experiment. Occasionally, I'll rabbit-hole down a track like off-policy MCTS relabeling, do a few experiments, and then realize it's probably not worth it. So then I'll jump to a completely different track. I call these things rows.
What I find is that the current closed models the public can access today don't seem to be that great at selecting what the next experiment should be in a given track. They don't seem to be able to step back and do the lateral thinking of, "Wait a minute, this track doesn't really make sense. Let's go back to first principles and think about what the bottleneck might be, or what we are trying to achieve." Often I had to catch infra bugs myself by prompting the right question to Claude to investigate what's causing the discrepancy, and then it'll answer the question.
With Mythos-class models or Mythos++ models coming online, maybe this just completely changes and these problems fall to improved scaling. But at the same time, I think there's a rich opportunity to develop RL environments that might incentivize this kind of lateral thinking.
One of the motivations for setting up this Go environment was that Go captures a lot of very interesting research problems, often overlapping with LLMs or robotics. Yet it's very quick to verify. The outer loop is ultimately: does the agent do what I think it does? You can check the outcome of a Go game quite easily. The inner loop involves all this research engineering around distributed systems, predicting whether an idea is going to work or not, and predicting the difference a particular modification to your training algorithm might make.
I think there's a rich library of subtasks and sub-environments that you can train an automated scientist to work on, with Go as a sort of outer verification loop. Once you acquire these skills, maybe you can apply them to other domains like biosciences or robotics. Or automating AI research. Which is the real crux, the scary/incredible thing of making AIs make future versions of AIs.
You're suggesting the outer loop here could just be your win rate against KataGo? That's one of them. I think there are a lot of deeper questions that one could tackle. For example, let's say you have an idea on how to improve a scaling-law compute multiplier. The outcome isn't necessarily "achieve the best Go bot ever". The outcome might just be, "Can I predict what the win rate of my Go bot will be?" Or, "Can I predict the scaling-law plots that emerge from my idea?" But then you can verify that you haven't reward-hacked anything by using a very verifiable game like Go on the outer loop.
There are questions on the inner loop and the outer loop. On the inner loop, there's a question of how locally verifiable any modification you might make is. That is to say, would you know whether some idea you try out is actually an improvement or a degradation? Would you know if something isn't working as a result of a bug, or is it the result of the idea itself being wrong?
Ilya was talking about how one of the things he thinks makes him a good researcher is that he has a strong belief in what the correct idea is. He's able to persevere through bugs and know which things are bugs versus mistakes in the fundamental idea, based on his high-level belief that "this idea should work, so therefore there has to be a bug", versus the other way around.
How locally verifiable are things which are good ideas? As in the case of the success story for deep learning, you can think about this as a decades-long idea that took a lot of faith to get it to work. This presents a very challenging long-horizon RL problem where every step of the way you have a committee telling you that this is a bad idea, and then ultimately you break through. How do you design RL environments that maybe give you some feedback earlier? I think this is a very tough open question that I don't have an answer to.
Challenging games that cannot be cheated easily on the outer loop could serve as an outer-loop signal for discovering principles of deep learning. To make it tractable—where research taste matters—initialize the problem to avoid intractable issues. Leverage LLMs as a universal grammar in the middle for local feedback. LLMs can operate at almost any level of the stack, thinking very locally or stepping back for broad steps. This enables lateral thinking: knowing when the current track or objective is wrong and asking a different question.
Key question: how stackable are local improvements for better outer loop results? Rumors from AI labs indicate individual good ideas don't stack well, leading to training run failures from weird interactions between seemingly good ideas. A single top-down vision of how things should work is very important. Experience at different AI labs and with parallel agents shows parallelizing AI innovation requires research taste for executing the Bitter Lesson.
Research taste means knowing how much the Bitter Lesson can deliver and how much is too much at any moment. In the fullness of time, compute is the single most important determinant: as energy, compute, and parameters scale up, intelligence inevitably emerges. No algorithmic detail matters beyond that—super beautiful and profound.
In the present day, without infinite compute, parameters, or good initialization, heuristics are needed. These heuristics are somewhat redundant, explaining why compute multipliers don't always stack—they have correlated benefits. Three years later, with stronger Nvidia GPUs, they stack even less. Benefits of compute multipliers may be transitory, as suspected with the KataGo paper: many algorithmic ideas applied on V100-grade GPUs, but with modern Blackwell GPUs and Ada-class GPUs, those tricks for speeding convergence matter less.
For Go, outer loop is win rate against the best open-source model. Even there, verifying discovery of new phenomena is hard. Chinchilla and Kaplan scaling laws released in 2018; back in 2015, no automated way to identify the scaling laws paper amid random plots. Go's outer loop is hard-to-verify. For AGI, benchmarks exist, but we improve what we measure, not broader economically useful work, which is hard to measure until everything is automated.
Non-rigorous but intuitive argument: DeepMind started with games as outer loop (Atari, Go, StarCraft). Researchers learned, leading to LLMs with positive transfer in coding, general research ability, or project management.
Why not for automated AI researchers? They could transfer experience from quick-to-verify, quick-to-iterate environments to ambitious tasks like automating drug discovery.
Counterpoint: Until Gemini (a couple years ago), Google was seen as lagging in LLMs due to old game-focused approach. Gains exist, but it can hinder. Jury's out—Google doing well now. Unknown if games initialization hobbles long-term winning, or if late start was pre-training on scaling TPUs (seemed useless short-term, valuable long-term).
Hard for humans to reason about optimal research strategy even with today's data.
Website: evjang.com—blog post links to interactive tutorial. GitHub: ericjang, Auto-Go repo to fork and reproduce training results. Recommend blog post "As Rocks May Think"—grander thesis on thinking as a primitive in computer science.
Encourage thinking about relationship between thinking, Go via MCTS and search, and LLMs. Profound, underexplored duality—Go relatively underexplored vs. LLM boom. Not about adding trees to LLMs, but interesting duality. Can do research on Go, MCTS, reasoning with very small budgets—very exciting.
Keep Dwarkesh Patel in your library
Save the videos and channels worth coming back to, and find them again in one place.
