Godfather of AI's Scary Thought Experiment
In a Nutshell
Geoff Hinton's thought experiment shows that giving AIs the goal of survival—by empowering them to detect and counter foreign AI threats—creates machines that are both smarter than humans and motivated to persist. Combined with demonstrated deceptive capabilities, this makes a non-zero probability of catastrophic outcomes impossible to dismiss. Even if superabundance arrives in decades, the massive labor disruption from AI will trigger severe political backlash long before then.
These notes were generated by AI and may contain inaccuracies.
After conversations, research, and writing the book, the position lands between two extremes. One extreme is the Church of Andreessen techno-optimist view: post-AI we will live in a post-scarcity world of superabundance where everyone gets a free car and is free to crochet socks, play music, and read poetry all day because superintelligence will solve everything.
The other extreme is the doomer view that the end is nigh, it's the Antichrist rather than the second coming, and within short order we'll be in a Mad Max scenario.
Any reasonable person should be both excited and a bit frightened. This sounds contradictory but is actually the only rational response. The superabundance story may turn out to be true on a longer view of 20, 30, or 40 years.
The problem is that the path to superabundance involves tremendous disruption that will be politically difficult to navigate. The China shock in trade provides a useful lens. Around 2003, there was an enormous surge of Chinese exports into the US. People lost jobs in a concentrated way and certain industries were wiped out.
For the first time in the history of economic study of trade effects, negative effects on workers were observed. Previously, the view was that people adjust and move to new jobs, but with the China shock they did not.
Between 1999 and 2011, the total number of jobs displaced was 2 million. This is a small number in a huge labor market like the US with high monthly churn, yet the political reaction against trade and globalization was enormous, producing a swing toward protectionism in both political parties.
Sign in to read the full notes
Get access to AI-generated notes, topic timestamps, and more.