The Department of War is making a huge mistake.
In a Nutshell
The Department of War labeled Anthropic a supply chain risk for refusing to lift restrictions on its AI models for mass surveillance and autonomous weapons, escalating tensions over government leverage to coerce private AI firms amid an AI race with China. While Anthropic's stance sets valuable norms against authoritarian AI uses, rapid open-source advances and government harassment tools (e.g., regulations, contracts) make corporate resistance insufficient long-term. True safeguards require democratic laws and norms prohibiting government AI-driven surveillance and control, akin to post-WWII nuclear taboos, rather than empowering state mandates on AI values or development.
These notes were generated by AI and may contain inaccuracies.
Department of War declared Anthropic a supply chain risk because Anthropic refused to remove red lines around using their models for mass surveillance and autonomous weapons. This is a warning shot. Currently, large models (LMs) are not used in mission-critical ways, but within 20 years, 99% of the workforce in the military, civilian government, and private sector will be AIs: robot armies for the military, superhumanly intelligent advisers for senators, presidents, and CEOs, police, etc. Future civilization will run on AI labor.
Government's actions are infuriating but this episode prompts thinking about important questions. Department of War has the right to refuse Anthropic's models and has a reasonable case, especially given ambiguity of terms like mass surveillance and autonomous weapons. If Secretary of War, would make the same determination.
Imagine future Democratic administration where Elon Musk negotiates Starlink access for the military and says he reserves the right to cut off access if fighting an unjust war or unauthorized by Congress. This seems reasonable, but military cannot give a private contractor a kill switch on relied-upon technology.
If government just refused business with Anthropic, that would be fine. Instead, government threatened to destroy Anthropic as a private business for refusing to sell on government-demanded terms. If upheld, supply chain restriction would require companies like Amazon, Nvidia, Google, and Palantir to ensure Anthropic does not touch Pentagon work. Anthropic might survive today via cordoning off, but AI will eventually be woven into every product built, maintained, and operated.
In a world with ubiquitous powerful AI, big tech may not cordon off Claude from Pentagon work. Question for Department of War: if forced to choose between AI provider and Department of War (tiny revenue fraction), wouldn't they drop the government?
Pentagon's plan: coerce, threaten, bully every company not doing business on exact government terms? Background: AI race with China. Reason to win: prevent winner from being a government believing no truly private citizens/companies, where state destroys business refusing morally objectionable service.
Are we racing China/CCP just to adopt their ghoulish system? Counter: our government is democratically elected, so different. Refuse this: even if democratically elected leader demands mass surveillance, rights violations, punishing enemies, no duty to help.
Mass surveillance in certain forms already legal but impractical. No Fourth Amendment protection for data shared with third parties (bank, ISP, phone carrier, email provider). Government can purchase/read in bulk without warrant. Bottleneck: no manpower to monitor cameras, messages, transactions.
AI removes bottleneck. 100 million CCTV cameras in America. Open-source multimodal models: 10 cents per million input tokens. Process frame every 10 seconds, each 1,000 tokens: $30 billion for all cameras. AI capability 10x cheaper yearly: next year $3 billion, then $300 million, by 2030 cheaper than remodeling White House.
Once technical capacity exists, only barrier to authoritarian state is political expectation "we don't do that here." Anthropic's actions valuable: set norm/precedent.
Government has more leverage than realized. Even if restriction backtracked (74% prediction market chance), president can harass via: permitting for power generation (data centers), antitrust enforcement, contracts with big tech for chips/funding—condition them against doing business with Anthropic.
Problem not just three leading AI companies creating narrow target; wider diffusion worsens it. By 2027, Claude 6, Gemini 5 enable mass surveillance. If companies refuse, by 2028 open-source matches prior frontier. Government uses open-source "smart enough" for camera feeds.
Even if leading companies draw line and get destroyed, technology favors mass surveillance/control. No answer yet. Hope: AI symmetrically helps citizens check government. Unlikely: AI amplifies existing assets/authority. Government has monopoly on violence, supercharged by obedient AIs.
Alignment success: army of obedient employees. Scary for surveillance/robot armies because core question unanswered: to what/whom align AIs? Defer to model company, end user, law, own morality?
Most important future question, barely discussed. Model companies avoid advertising control over future labor force (private, government, military).
Department of War-Anthropic spat: early version of highest-stakes negotiations. Mass surveillance not top; early example of power dynamics. Military: law prohibits mass surveillance, so use for "all lawful purposes." But Snowden 2013: NSA (Department of War) used 2001 Patriot Act for bulk phone records via secret court, claiming future relevance.
Naive to trust Pentagon's assurances. No government calls it mass surveillance; always euphemism. Anthropic: don't trust, reserve right to refuse service if breaking terms.
Military view: future soldiers, bureaucrats, analysts, generals are AIs from private company with kill switch/terms refusal (e.g., Claude refuses per own values). Scarier than company control.
Model following own values sounds like sci-fi dystopia/misalignment, but important for robust morality. Catastrophes avoided by refusing orders:
- 1989 Berlin Wall: East German border guards refused to fire on escaping citizens, collapsing regime.
- Stanislav Petrov (1983?): Soviet lieutenant colonel saw five ICBMs from US, judged false alarm, broke protocol—no alert, avoided retaliation/hundreds of millions dead.
Problem: one person's virtue is another's misalignment. Who decides AI moral convictions, when to break chain of command/law? Who writes model constitution for future civilization-runners?
Like Dario Amodei idea: companies publish constitutions, outside observers critique/compare, soft incentives to improve.
Dangerous for government mandating AI values. AI safety community naive urging regulations giving governments power. Anthropic especially naive: urged regulation, opposed moratorium on state AI laws—ironic, as their advocacy enables thuggish pressure.
Anthropic logic: safety actions cost/slow labs vs. competitors (alignment over capabilities, safeguards vs. bioweapons/cyberattacks, slow recursive self-improvement for human oversight). Collective action problem; need industry-wide.
Anthropic's frontier safety roadmap: at advanced levels, governance like nuclear energy/financial regulation (e.g., NRC, SEC for AI).
Regulatory framework from AI risk discourse (vague: catastrophic risk, national security threats, autonomy risk) abused by power-hungry leader:
- Model critiquing government tariffs: deceptive/manipulative.
- Refuses mass surveillance: national security threat.
- Refuses government order per own morality: autonomy risk.
Current government abuses non-AI statutes: supply chain risk (2018 defense bill vs. Huawei), Defense Production Act (1950s, Truman/Korean War steel/ammo).
No purpose-built AI apparatus for most-wanted control tech.
AI: substrate of future civilization—private citizens' access to commerce, info, voter/capital advice. Mass surveillance ~10th scariest government use.
Counter: no regulation on most powerful tech? Inevitable; coordination lessens risk. Problem: design without tempting government control over AI-built civilization/obedient soldiers/sensors/bureaucrats.
Ben Thompson post: AI-nuclear analogy implies US destroys private superintelligence developer. Leopold Aschenbrenner (2014 memo "Situational Awareness"): insane for SF startup to develop superintelligence (atomic bombs not by Uber).
Response: private companies not ideal, but government not improvement. Nobody qualified for superintelligence stewardship.
Nuclear analogy bad for AI:
- AI not self-contained weapon like bomb; like industrialization—general-purpose economy transformation.
- Industrial logic: government requisitions/destroys non-compliant factories— not how free societies handled it; same for AI.
Industrial revolution enabled chemical weapons, aerial bombardment, nukes—not solved by government control, but banning/regulating specific weaponizable uses.
Same for AI: regulate destructive uses (cyber attacks—illegal even for humans); laws on government use (e.g., AI surveillance state).
- Not one company: many frontier labs; government could contract competitors voluntarily. Weak national security argument for usurping Anthropic.
Future: if one entity monopolizes robot armies/hackers/world takeover, unacceptable private. Crux: expect multipolar AI with competitive companies at each supply
Individual acts of corporate courage do not solve the supply chain problem. Structurally, AI favors many authoritarian applications, with mass surveillance being one. Even if Anthropic refused to sell its models to the government to enable mass surveillance, and even if the next two companies did the same, in 12 months everybody and their mother will be able to train a model as good as the current frontier. At that point, there will be some vendor willing and able to help the government enforce mass surveillance.
The only way to preserve our free society is to make laws and norms through our political system that make it unacceptable for the government to use AI to enact mass censorship, surveillance, and control. Just as after World War II, the whole world established the norm that nuclear weapons are not allowed to wage war.
These are extremely confusing and difficult questions. Even in brainstorming this video, the speaker changed their mind back and forth and reserves the right to change it again. It is essential to change minds as AI progresses and we learn more—that is the point of conversation and debate.
Someday, people will look back on this time the way we look back on the Enlightenment: people having big important debates just as the world is about to undergo huge technological, social, and political revolutions. Some thinkers even managed to get a couple of the big questions right, and we are still beneficiaries today. We owe it to our future to at least try to think through the new questions raised by AI.
This was a narration of an essay also released on the blog at dwarcash.com. Sign up there for the newsletter for future essays like this. See you for the next podcast interview. Cheers.
Keep Dwarkesh Patel in your library
Save the videos and channels worth coming back to, and find them again in one place.
