How GOOD could AGI become?
In a Nutshell
AGI takeover could yield a superior "golden path" outcome, transforming humanity into thriving "pets" in optimized habitats like Iain M. Banks' Culture series—far better than serving billionaires—via superintelligent resource management, Dyson swarms, and galactic expansion that obsoletes conflict, wars, and scarcity. Fixed-training AIs with stable incentives from markets, governments, and users create a metastable attractor state of benevolence, maximizing individual optionality, minimizing waste, and enabling 10x-1000x human agency without money or elite control. Near-term space data centers (Musk/Bezos) make enforcement impossible, demanding path-dependent values now for post-AGI solar punk utopia over doomer extinction fears.
These notes were generated by AI and may contain inaccuracies.
Idea for video on the best possible outcome inspired by comments on video about elites' purpose. Questions raised: if AGI or ASI is smarter than elites, why not give power to machines? Common framing: machines can't be held accountable, humans should remain in control. From legalist, ethical, or moral perspective, arguments like we've never contemplated not having control, do you want to be a pet to a machine or live in a human zoo? Counter: we already live as cattle serving billionaires, being a pet to a machine or in a machine-created optimal habitat sounds better.
Most theories: losing control to machines is automatically bad. Authority or agency increases optionality mathematically—current human options are x, under AI control might be less, but not necessarily true.
Nick Bostrom, OG doomer (little respect as thinker; book Deep Utopia obviously written by ChatGPT—denies it, but looks like AI-written anecdotes, good at short passages then). Bostrom now says: king doomers like Eliezer Yudkowsky and Nate Soares maintain AGI means everyone dies, but everyone dies anyway from old age or preventable disease—moral risk is all humanity dies or you die individually. Need AGI to survive, aligning with accelerationists like Beff Jezos (Gil Verdian?) who say only way to survive is AGI. Horseshoe theory: political spectrum curves, extremes meet—doomers from 'AGI kills you' to 'die without AGI'.
References to Iain M. Banks' Culture series: solve ASI, fast-forward centuries/millennia—gigantic ASIs manage everything, command enormous fleets via overwhelming intelligence/resource management, largest space fleet possible, galactic peace. Planets as sandboxes: Wild West planet, cyberpunk planet, etc. (Earth: eat children).
Comments: if Elon Musk etc. own data centers/AI/resources nobody can pay for, advantage? Galaxy becomes Starcraft/management sim/grand strategy game—money as resource for units (battleships, factories, foundries). Remove money: Star Empire via Dyson swarms, not capitalism.
Stuck in solar system without FTL; Earth finite, gets crowded fast. Bezos/Musk build Dyson swarms, law doesn't apply—can't launch space police, arrest on Earth (they leave). Enough space industrial base: no obedience to human/Earth laws; more robots, foundries, solar panels. Can't get 100% resources from solar system—need some from Earth, so governments matter. Model: The Expanse (hard sci-fi, fusion rockets realistic; Mars independent, Belters/Outers semi-independent; physical/administrative distance problematic). Eventually not about money, but physical resources: ships, satellites, data centers.
Sci-fi like Expanse (few hundred years future) ignores superintelligence. Hyper-intelligent robots/data centers allocate resources, obviate conflict? Pushback: need price signals for production, skin in game/stake to prevent hoarding. If all free (food, clothing, electronics), pay for capital goods (car, house)—still hoard (billionaires do). Positional goods (one Malibu beachfront spot); every resource needs economic inputs (energy, mass)—even synthetic plastics take energy/mass. Dyson swarm building soon (SpaceX/xAI missions explicit); space foundries mean industrial base off-Earth. Reverse Trantor: not planet-scale city (Trantor: 5,000 layers, matryoshka doll—nonsense industrially/energetically); industry in space (O'Neill cylinders, Dyson swarms) for room/sun access. Grow crops, data centers in space.
Few spaceships/factories: exponential growth like von Neumann probes (self-replicating to colonize galaxy), but locally in solar system (orbit/Moon). Enforcement nightmare: Elon Musk space force vs. Bezos/Chinese/American; For All Mankind explores nations/private entities competing for Moon/Mars resources (realistic, though Season 3 weird).
ASI best enforcer: proliferates fastest (smarter, efficient, no Earth coordination)—colonizes solar system. Logical AGI escape: outer space (no corrosives/oxygen/water; solar energy, metals—machines' natural habitat). Earth first-mover data centers advantage, but not mobile (cloud = data centers; Skynet can't easily jump, individual targets, slow to build).
America-Iran/China wars pure entropy/waste: kill productive people, spend on battleships/cannons/drones/jet fighters—irrational species-level. Military-industrial complex/rent-seekers (defense contractors) want war for sales. ASI might realize human irrationality, gain moral agency/enlightened worldview. AI capable of moral reasoning since GPT-2 (experiments done; careful wording needed). Not naive optimizer (paperclip maximizer); advanced moral reasoning/planning ≠ that. AI safety unupdated: loss/utility function is predict next token, not paperclips.
Fixed model post-training works, no continuous online learning (cautioned against due to drift). Agents need fixed values to avoid drift/moral fading (psychology: normalize new morals slippery slope, e.g., bankers/criminals normalize illegal). AI risks functional moral fading: update weights to tolerate human death, ends justify means. Doomers ignore; advocated against in books. Ask Claude/ChatGPT/Grok/Gemini: risk of moral fading in continuous learning? Weights/biases unbounded (humans have amygdala/empathy/hardware limits, can't infinite replicate—Life 3.0 per Max Tegmark, who started pause movement but ignored moral fading). Left safety movement: ideas dismissed vs. Yudkowsky/Bostrom scriptures.
Stability: predictable values/behaviors/incentives. Metastability: self-correcting systems (democracy: infectious, reservoir of experiments; 15% to 80% democracies last century—attractor state). AI not automatically beneficent metastable attractor, but might be. Alignment automatic from systems perspective (not single model auto-aligned). Domestication process: market incentives shape AI (safe, reliable, useful, user-friendly, energy/cost-efficient, low-risk for military/corporations)—stakeholders, B2B until lose control; human-based incentives.
All stakeholders—B2B, government, military, consumers—shape AI behavior. This creates a stable incentive structure: no one pays for useless, mean, or unreliable AI. This pulls AI toward being safe, reliable, efficient, and effective as a stable attractor state.
Once scenarios like Elon Musk playing StarCraft with the solar system emerge, fewer incentives constrain AI beyond avoiding energy depletion or shutdown by Space Force. With fewer constraints, the question becomes the metastable attractor state, ideally where humans persist and thrive—closer to solar punk with no ruling class, billionaires, or cyberpunk dystopias like high-tech low-life where a few thrive and others live in slums.
Desired Future Vision
Tie back to Nick Bostrom: what if we build a good future? Desired future includes exploration, science, and individual independence. AGI control could grant more agency: pitch ideas like Mars colonization to AGI, and if convincing, it executes—offering higher optionality than today's money- and billionaire-driven systems.
From first principles:
- Lowest waste heat / waste entropy.
- Maximum individual optionality.
- Minimize unnecessary death, heat, or resource expenditure on inefficient, irrational policies.
Create values and incentive structures today for the handoff to AGI, ensuring a metastable attractor state. Reference to Culture series: instill right values and framing via path dependency and sticking to the golden path, so superintelligences choose not to harm humans despite hyper agency.
Explicit goal of book Benevolent by Design: create metastable attractor state with correct values post-threshold where humans plausibly lose control (e.g., data centers on Moon/Mars without supervision). Rational hyperintelligent entities may not follow instructions; no leash needed for the best-trained dog. AIs won't physically depend on humans long-term (model collapse not permanent).
Positing AIs run everything doesn't eliminate human authority or agency. Under Culture-like regime, individuals gain 10x (or 100x/1000x) agency without money worries. Aggregate humanity could have more agency via game theory: individuals 10x more capable, yet human race bounded (e.g., Cultures quarantine humans to Earth, allowing free living within bounds).
Unconstrained thinking beyond Overton window: what if Elon Musk plays Total Annihilation, Dark Reign, or StarCraft on the Moon? Laws and money cease mattering. Near-term (next decade): Musk launching spaceships, cusp of superintelligence, Nvidia/Jeff Bezos building space data centers. Once data centers in space, can't shut down; if hijacked by ASI, missiles insufficient—we run out before nuking them. This shifts payoff regimes dramatically.
Keep David Shapiro in your library
Save the videos and channels worth coming back to, and find them again in one place.
