How Intelligent Is AI, Really?
These notes were generated by AI and may contain inaccuracies.
Greg Camrad, president of the ARC Prize Foundation, a tech-forward nonprofit, aims to pull forward open progress towards systems that can generalize like humans.
François Chollet defines intelligence as the ability to learn new things much more efficiently. ARC Prize uses an opinionated definition from Chollet's 2019 paper 'On the Measure of Intelligence': intelligence is the ability to learn new things, not just scoring high on tests like SAT or hard math problems.
AI excels at chess, Go, and self-driving (superhuman levels), but struggles to learn different skills.
Chollet proposed the ARC AGI benchmark (initially ARC) to test learning new things over hours, days, or a lifetime. Both humans and machines can take it.
Unlike benchmarks like MMLU, MMLU plus, or Humanity's Last Exam (PhD++ problems going superhuman), ARC ensures normal people can solve tasks.
Context: Pre-2024 LLMs performed terribly on ARC-AGI-1. Large language models scored low despite pre-training.
In 2019, Chollet launched ARC with 800 tasks he created himself.
By 2024, GPT-4 base model (no reasoning) scored 4-5%. o1 and o1-preview jumped to 21%, highlighting the transformational impact of the reasoning paradigm.
Now, labs like XAI, OpenAI use ARC-AGI in model releases: Grok 4, Gemini 3 Pro, DeepThink, Opus 45 (past 12 months).
Excited that frontier labs adopt ARC-AGI, endorsing it in releases, but mindful of vanity metrics. Primary mission: inspire small teams, individual researchers for open AGI progress.
Wearing 'romantic pursuit of general intelligence' hat vs. economic/product hat.
RL environments allow scoring well on specific benchmarks/domains but like whack-a-mole; not generalizable to novel future problems (ARC has hidden test set). Humans don't need custom environments. Prefers investment in generalizing systems over short-term RL gains.
ARC-AGI-1 (2019): 800 static tasks by Chollet.
ARC-AGI-2 (March 2025): Deeper, upgraded static benchmark.
ARC-AGI-3 (next year): Interactive, ~150 video game-like environments (action-response). No instructions, no English/words/symbols. AI must take actions, observe responses, infer goal.
Fun games; test on general public (accountants, Uber drivers). 10 people per game; exclude if below minimum solvability threshold by regular humans.
Contrasts benchmarks that escalate difficulty; ARC-3 solvable by humans but not AI signals missing capabilities, spurring new research.
Intelligence measured by accuracy, data efficiency, energy efficiency, and sample efficiency (actions/turns), calibrated to human baselines.
Time is arbitrary (more compute reduces it). Key: training data needed, energy to execute.
Humans: Known data points and brain energy consumption per task.
ARC-3: Turn-based games; count human vs. AI actions to solve. Normalizes to average human performance; no brute-force (unlike 2016 Atari: millions/billions of frames/actions).
Chollet: Solving ARC-AGI necessary but not sufficient for AGI.
ARC-1/2: Authoritative source of generalization.
ARC-3: Most authoritative evidence to date of generalization.
If solved tomorrow: Analyze system, find failure points, guide towards proper AGI. ARC Prize positions to declare AGI when ready; would converse with the team.
Keep Y Combinator in your library
Save the videos and channels worth coming back to, and find them again in one place.
