The data black hole at the center of AI

Name: The data black hole at the center of AI
Uploaded: 2026-06-19T17:17:05.000Z
Duration: 11 min 57 s
Channel: Dwarkesh Patel

Dwarkesh PatelJune 19, 202611m

Topics9

Sample Efficiency as a Definition of Intelligence0:00 Task-Specific Nature of Human Expert Data1:04 Data as the Primary Driver of Progress2:34 Comparative Scale of Human vs. Model Data Requirements3:30 Addressing the Evolution Objection4:31 Addressing the Multimodal Data Objection6:03 Addressing the Scaling Objection6:30 Implications for White-Collar Automation8:46 Limits of Pre-Training for Out-of-Distribution Work10:34

In a Nutshell

Frontier AI progress stems almost entirely from scaling massive, task-specific human expert datasets and synthetic RL data rather than improving sample efficiency. Models require orders of magnitude more data than humans—trillions of tokens versus a human’s ~200 million—to reach competence, and even infinite parameter scaling would close only a small fraction of this gap. White-collar automation is feasible only for in-distribution tasks, while out-of-distribution work and true intelligence explosions depend on solving the unsolved sample-efficiency bottleneck.

AI-Generated Notes

These notes were generated by AI and may contain inaccuracies.

One definition of intelligence is sample efficiency: how much data is needed in a given domain to operate fluently and competently. Little progress has been made in improving sample efficiency over recent years. Instead, gains have come from dramatically widening and improving the data distribution. The primary driver of AI improvement has been adding more and better data while scaling the compute required to develop that data.

RL functions as synthetic data generation by applying large amounts of compute against a verifier or rubric (such as an LLM judge) to identify high-quality data, then training the model to predict those correct rollouts in the same manner as next-token prediction on internet text.

For this process to succeed, the model must possess some prior probability of generating the correct solution. This requirement explains the need for large volumes of human expert trajectories across every field and skill the model is intended to master.

Human expert data is highly task-specific and bespoke. Job listings on platforms such as Mercor and Surge illustrate this specificity, including roles for Word specialists converting legacy documents into polished files, legal experts producing realistic M&A diligence reports or securities filings, and management consultants creating template market research documents.

Beyond domain specificity, the volume of data required is substantial. Each skill involves hundreds of human experts generating example completions, writing rubrics, and explaining chain-of-thought reasoning. The data industry supplying these expert labels and the RL environments in which these skills are developed generates billions of dollars in annual revenue, with projections reaching deca-billions.

The scale difference is stark: a human student might solve a textbook problem once or twice, whereas models using GRPO generate hundreds to thousands of rollouts per task to address the credit assignment problem. The correct framing is not that these models have learned skills like humans, but that they are constructed like a Frankenstein's monster assembled from a billion carefully constructed example grafts.

Epoch AI reported that open models lag state-of-the-art frontier models by four months. The relative ease with which open-source efforts catch up within months stems from data being the real driver of progress. Data can be distilled from public APIs, whereas hyperparameters, training tricks, and architectural optimizations cannot be similarly extracted. If the latter factors drove most progress, catching up would be substantially more difficult.

The quantity of data used to train these models greatly exceeds human lifetime experience. Frontier models are trained on tens to hundreds of trillions of tokens compared to the roughly 200 million tokens a human encounters from birth to adulthood (based on an estimate of 2,000 words per hour).

Sign in to read the full notes

Get access to AI-generated notes, topic timestamps, and more.

The data black hole at the center of AI

Sample Efficiency as a Definition of Intelligence00:00

Task-Specific Nature of Human Expert Data01:04

Sign in to read the full notes