Beyond Bigger Models: Recursion As The Next Scaling Law In AI
In a Nutshell
Hierarchical Reasoning Models (HRM) and Tiny Recursive Models (TRM) enable small models (7-28M parameters) to outperform massive LLMs like o3 on ARC Prize (up to 87% on ARC-1) by using recursion for deep reasoning at inference time, bypassing transformer layer limits and backprop-through-time issues via truncated BPTT (t=1), fixed-point iteration, and EM-style latent state updates. Unlike one-shot LLMs or hacks like Chain-of-Thought/tool use (bounded by human data), these RNN-inspired models discover efficient algorithms from scratch on incompressible tasks like Sudoku/mazes, compressing reasoning into hidden states with outer refinement loops. Recursion scales compute without parameter bloat, suggesting hybrids of tiny recursive heads atop giant LLMs as the next AI frontier.
These notes were generated by AI and may contain inaccuracies.
Welcome back to another episode of Decoded. Today with YC visiting partner Francois Shaard discussing recursion to improve model's reasoning performance at inference time rather than making models bigger. Two papers from 2025: Hierarchical Reasoning Models (HRM) and Tiny Recursive Models (TRM).
Francois, thanks for joining. Tell us about these two models and what was interesting.
To set foundation, RNN is a model recursively called on itself. Belief that RNNs required for AGI. Peak RNN use until 2016 with Alex Graves' Neural Turing Machines keynote and adaptive computation time work. This was 10 years ago in era of LSTMs with attention, before attention invented depending on professors.
Limiting step on RNNs: backprop through time. Roll out model, approximate gradient, step back. As model bigger and more steps, accumulation of errors, noisier gradients, stops working. Vanishing or exploding gradients: input with 20 steps multiplies matrices 20 times. For context length million or billion, worse. Retain activations every step: need million copies of brain. Tricks like gradient checkpointing trade memory for wall clock time and compute.
Contrast with LLMs: at training, one-shot feedforward for every input. Transformer block takes all inputs in parallel, not iteratively. No storing tons of activations or giant vanishing gradients.
All happens in one shot with causal mask, lower triangle trick. Forward and backward in one shot, amazing for training wall clock despite high FLOPs and memory. Tradeoff: lose latent reasoning and compression in time direction. No compression in LM: retain entire Shakespeare novel to decode bit. RNNs compress in hidden state.
LLMs do reasoning in forward pass? Go back to GPT-2: next token prediction, val loss down, perplexity down, generates plausible Shakespeare.
To reason and solve hard problems: take unsorted lists, feed sorted lists, should learn sorting. Impossible in one shot: theoretical lower bound for comparison sort n log n steps. List 31 elements, transformer 30 layers, run out of steps for comparisons.
HRM and TRM use Sudoku, mazes, rolling sum as incompressible problems.
Sorting faster than n log n with external memory cache or tape, selectively putting things. Key LLM limitation: no external memory tape, lose performance possibilities.
Radix sort: depending on buckets, n log n to order n, can't less than n, touch all elements. Run out of transformer layers, out of chances.
Analogy to Turing machine: GPT-2 feedforward, one step input to outputs. Sudoku: 50 givens, provable only one possible, limited layers. At test time Turing complete, simulate all Turing computable functions, but training issue. No human-labeled traces for Millennium Prize problems.
HRM in lineage of RNNs. Bio-inspired: brain parts operate different frequencies, high frequency low hierarchy, low frequency high hierarchy.
Training: input X (Sudoku, maze, ARC prize). TL steps with lower level module L. Then TH steps with H. N outer refinement steps.
Three levels recursion: low level, high level, outer refinement. Same weights applied repeatedly, no changing weights.
Recurse LNet TL times, recurse TH times, outer refinement N times.
Intuition: variable scoping. Lowest function scoped variables ZL (carry, latent variable, hidden state). Recurse, pass ZL to higher level. Higher does iter, calls lower again. Outer loop refinement.
State-of-the-art on ARC Prize 1 and 2. 27 million parameter model, trained only on ARC Prize (~1000 puzzles), no pre-training, from tabula rasa. Outperforms o3 (0% on ARC Prize 1, ~70% on ARC Prize 1).
Backprop issue: HRM uses deep equilibrium learning (DEQ), fixed point iteration, not backprop through all recursion like Alex Graves (Neural Turing Machines, adaptive compute time, differential neural computers).
DEQ: take batch, forward, loss, backprop, update weights, normally new batch. HRM does 16 times on same batch. Residuals get smaller.
RNN hidden states start zeros. After loops TL/TH, backprop once through two modules, stop-grad, don't recurse back. Huge residual, don't reset ZL/ZH, iterate from different point in carry space. Like different batch every time, same X.
16 recursions construct mini-batch from different memory states, across hidden/carry memory.
Math follows DEQ if delta ZL/ZH -> 0, but doesn't. Alexia shows backprop through deep recursion improves performance more.
Bio-plausibility inspires but often veer to bio-implausible that works better, runs better on GPU.
ML overlaps neuroscience, brain efficient. Neural networks model neurons, activation functions biological.
Use bio-plausibility to inspire ideas, but not bounded by it.
Compelling view: automata theory, data structures/algorithms. Memory cache useful for efficient algorithms. Hidden states/carry akin to Turing tape or radix sort memory bank, train model to use intelligently in single forward pass for efficient operation.
CoT and tool use get beyond GPT-2 limits. Infinite unsorted/sorted lists with CoT teaches every step, becomes Turing machine at test time.
Cheaper: teach Python sort function, call it, no backprop.
Hacks bounded by human knowledge. Chain of thought doesn't discover sorting from first principles, from training data. Tool use knows only trained functions (e.g., bubble sort).
Einstein test: rebuild physics from 1911. If only bubble sort data, only does bubble sort poorly.
Recursion in LLM: chain of thought in token space of outputs, not inherent model. Feedforward one-shot, hack by reading own outputs, upper bounded by training data.
Both CoT/tool use bounded by human knowledge. Outside, fails.
Discrete vs latent: LLM carry snaps to discrete tokens. RNNs continuous latent space, higher dimensional, more expressive, but backprop through time inhibits training.
Number one: outer refinement loop scales. Sapien authors innovations. Constantine at Fronton's company India breakdown on YouTube: outer refinement loops main reason works well. Alexia scales it up, gets rid of other stuff.
TRM: outer refinement loop great. Truncated backprop through time to t=1 sufficient (HRM finding). Counterintuitive.
HRM one call hnet/lnet, TRM one full recursion loop. 16 times, backprop one time. Fixed point iteration, gradient every step, batch across carry space works.
That part kept. Changed: double layer
TRM collapses the double layer of higher-order and lower-order thinking into a single network. The key is having low-level and high-level features in the same network with weight sharing between lnet and hnet, now just called net. It uses one transformer layer instead of four as in Sapient, while keeping ZL and Zh distinct. TRM ablates two separate networks versus one, finding the single network performs best. The nomenclature in TRM uses X and Y (confusing; Y is latent space like Z, not a label), but Sapient's Zh and ZL is cleaner.
Recursion provides compute depth without parameter depth, avoiding hundreds or thousands of transformer layers and massive parameters. The optimization resembles an iterative expectation maximization algorithm.
Update ZL conditioned on input X and previous Zh (t-1), iterating ZL updates multiple times. Then update Z conditioned only on ZL. ZL acts as local scoped variables that overwrite and update. Z is a candidate answer, a proposed latent embedding one MLP lookup from the true answer. This EM process maximizes probability of correct information in memory conditioned on output, and right output conditioned on memory, in parallel. It learns a recursive method to store right information in local memory and output correctly.
Sudoku is an incomplete puzzle where you guess one or two cells at a time based on available info; it's incompressible without random guessing in high combinatorial space. ZL tries options, computes, thinks, proposes; Z conditions on it, fills in, advancing the puzzle. Training maximizes strategies for saving info that lead to correct outputs without chain of thought. It discovers solutions without teacher forcing via chain of thought, as if solving Sudoku without prior knowledge.
Start with Z and ZL as zeros. Embed raw input x to x (e.g., maze state). Use no_grad to avoid backprop through time. For simplicity, hit ZL t-low times, then Z once modulo t-low. Update ZL conditioned on Z and X, then Z on ZL (EM-style). Freeze weights past no_grad line. Hit ln and hnet (now net) one more time. Two recursions shown; third in train/test loops.
- Use 2 recursions (sufficient; higher doesn't help per Constantine).
- n_sub (deep supervision, adder refinement steps) recurses n_sub times in training; m_test at test time (often same hyperparam).
- Train on 16 recursions, test on 1 retains ~70% performance; often redundant compute.
Backprop through two small parts, step, zero gradients, but do not update Z/ZL. Pass previous Z/ZL to next iteration (different latent space, key mini-batch construction through memory space). At test time: outer refinement loop (train recursion important, test less so), inner HRM with two loops.
Collapse lnet/hnet to single net (1 transformer layer vs 4). Alexia shows deeper layers don't help; on Sudoku, MLP outperformed transformer (0 on maze). Weight sharing. Backprop through one full latent recursion step (vs two before). Do 6 inner loops, one more, then deep recursion n_sub times with no_grad/detach, followed by latent recursion. Otherwise identical training/test.
HRM: 28M to 7M parameters, 70% to 87% on ARC Prize 1, strong on ARC Prize 2. Smaller model outperforms due to recursion. Melanie Mitchell's book: scaling bigger sufficient but not necessary; recursion also sufficient not necessary.
Excited for combining big models + recursion, avoiding backprop through time limits (Alexia limited by last step memory). Big model + lots of recursion + alternative to BPTT yields crazy results.
Recursion important, not going away (e.g., Google recursion language models). Add refinement loop (truncated BPTT t=1) powerful, underexplored. Tiny 7M recursive models solve what 100B+ trillion-param internet-trained models can't. Combine TRM/HRM benefits with giant LLMs (possibly already in Gemini) for takeoff.
TRMs/HRMs task-specific (Sudoku model can't do ARC Prize without retraining); LLMs general-purpose with fine-tuning/ICL. Overlap: general-purpose agents with next-token prediction + complex reasoning via efficient recursive architectures.
LLMs excel at embedding spaces but reason through token space. Future: map tokens/pixels to semantic latent space, then apply tiny recursive reasoning models trained there.
Keep Y Combinator in your library
Save the videos and channels worth coming back to, and find them again in one place.
