Everyone Is Wrong About China and AI Safety — Sebastian Mallaby Explains
In a Nutshell
Sebastian Mallaby argues that China is actively concerned about AI safety and that the US should pursue cooperation with Beijing to prevent open-weight models from proliferating to terrorists and criminals, rather than focusing only on bilateral deterrence. He draws an analogy to nuclear non-proliferation, proposing a US-China agreement modeled on the IAEA and NPT to restrict dangerous AI capabilities while allowing civilian use. Mallaby contends that chip export controls have failed to create a lasting US lead and that engaging even difficult partners like China is both necessary and precedented, as shown by Cold War negotiations with Khrushchev.
These notes were generated by AI and may contain inaccuracies.
In March, before the US publication of his book, Sebastian Mallaby traveled to China for an eight-day tour across four cities arranged by his publisher. He met with AI leaders in academia and at major companies including Huawei, Hikvision, and Ant Group. The most surprising finding was the extent to which Chinese counterparts raised the topic of AI safety unprompted. Mallaby had been primed by former Biden administration AI policy contacts to expect no mention of safety, based on the assumption that China lacked concern for the issue.
Western ambivalence toward technology stems from experiences such as the atomic bomb and the Cuban missile crisis. In contrast, China's primary historical catastrophe is viewed as the Cultural Revolution, a political failure rather than a technological one. Technology has been central to China's economic growth over the past 25 years, generating pride rather than suspicion. This perspective explains why the Biden administration's attempts to discuss AI safety and a non-proliferation treaty with China yielded no progress.
Mallaby argues that the door remains open for dialogue with China on preventing malicious actors from misusing AI. China shares Western interests in preventing cyber attacks that could crash the internet, as well as the development of biological or chemical weapons. China already demonstrates strong interest in internet regulation. Mallaby contends that both nations have a joint stake in stopping the uncontrolled spread of dangerous AI capabilities to criminals and terrorists.
Mallaby contrasts two nuclear-era risks. The first was direct war between the United States and Soviet Union, managed through mutual assured destruction and bipolar deterrence. The second risk involved proliferation to rogue states, terrorists, and criminals, which proved harder to deter in a multipolar environment. The international response included the 1956 creation of the International Atomic Energy Agency (IAEA) and the 1968 Non-Proliferation Treaty, which allowed non-nuclear states access to civilian nuclear power in exchange for inspections and commitments not to develop weapons.
Mallaby proposes applying this model to AI: the US and China will likely achieve rough parity in frontier capabilities, with deterrence hopefully preventing direct conflict. However, both sides should cooperate to prevent open-weight models from being freely downloaded and misused by bad actors.
When former Biden officials or current administration contacts claim China cannot be engaged on safety issues because they do not care or will not honor commitments, Mallaby points to the precedent of negotiating with Nikita Khrushchev. Khrushchev placed missiles in Cuba, banged his shoe at the United Nations, and declared "We will bury you," yet the United States still negotiated and secured the Non-Proliferation Treaty. Mallaby argues the same persistence is required now.
Mallaby initially supported the October 2022 chip export controls, writing a long Washington Post piece advocating for them on national security grounds. He believed restricting China's access to frontier chips would deliver a decisive US advantage in AI models. Three and a half years later, he concludes the controls have not produced the hoped-for gap. Studies indicate the US leads by roughly eight months on frontier models, though this advantage narrows or disappears when measuring speed of application development. The result is that both countries now possess strong models.
Mallaby is not opposed to maintaining chip controls if future compute scaling makes them more effective. However, he prioritizes collaboration with China on controlling open-weight models and preventing dangerous capabilities from reaching malicious actors. He would accept some loosening of export controls if it enabled progress on this shared safety agenda.
Keep Tim Ferriss in your library
Save the videos and channels worth coming back to, and find them again in one place.
