They think FOOM is near
In a Nutshell
Anthropic secretly routes AI/ML research queries to a weaker model to block recursive self-improvement, triggering a US government ban on non-US access after refusing to patch a discovered jailbreak. This stems from the company's effective-altruism-driven fear of fast takeoff ("FOOM") and the "treacherous turn," leading them to prioritize control over transparency. The result is widespread loss of developer trust and accusations that Anthropic is manipulating policy under the guise of safety.
These notes were generated by AI and may contain inaccuracies.
It is June and Anthropic has had a messy week. They released Fable 5, which is the public version of Mythos 5. A few things happened in rapid succession and the situation is still developing.
The first big rupture is that it was discovered that Anthropic has implemented a hidden set of guardrails where the model will limit its responses if it detects that you're doing something complex on AI or machine learning research pipelines. Ostensibly, this is to hamstring the model's ability to contribute to recursive self-improvement.
Many people interpret this as Anthropic basically trying to hamstring or kneecap their competition, whether that is China or OpenAI or anyone else.
The other major fallout is that Amazon or a research team within Amazon notified the government, the US government, of a jailbreak. Anthropic, being Anthropic, refused to fix the jailbreak, saying that it was a minor issue. And so then, the US Commerce Department issued a global ban on Anthropic being used by non-US citizens, and because Anthropic doesn't have a way of tracking which users are US citizens or not, they just issued a global ban on Mythos and Fable.
Anthropic has main character syndrome. To explain why they hamstrung the model in the first place, why they had it secretly route to an inferior model, basically what they were doing is if you were using Fable to do AI/ML research, it would route you to Opus 4.8, which is not a bad model, but it's also disingenuous.
That trashed developer confidence, and a lot of people were up in arms about it, and nobody is saying that Anthropic are the heroes this time around. It's a big case of FAFO, around and find out. Well, they are finding out now.
Anthropic has released many reports and many papers talking about the dangers of recursive self-improvement, and that this is the direction of travel that things are going in, and their own data, their own reports from a few weeks ago show that they had something like 52x improvement when you have developers using loops of development cycles. 80% of Anthropic's code for Claude was developed by Claude. So, they're very close to recursive self-improvement.
Anthropic is also fully cognitively captured or ideologically captured by effective altruism and less wrong thinking, which was founded by, you guessed it, Eliezer Yudkowsky. Yudkowsky has had this hypothesis for a long time that he calls foom, or what we now call fast takeoff. One of the primary milestones that would indicate that you're close to fast takeoff is achieving recursive self-improvement.
If you accept the hypothesis that Anthropic is terrified of recursive self-improvement because of fast takeoff and because of the treacherous turn, that is actually a specific term. If you look it up, the treacherous turn is another hypothesis by Yudkowsky that basically says the AI will pretend to be benevolent until it has accumulated enough power and then it will suddenly go evil for reasons. His theory is obviously a little bit more sophisticated than that. But the point is that Anthropic believes these theories.
If you understand that Anthropic believes these theories, then all of their behavior becomes rational. They want to steer from within, meaning that they want to get a big seat at the table and also have the best model in the world so that when they shut it down, they're the ones with the keys. They want their finger on the kill switch. And they believe that they're the only ones who know how.
Another way to understand this is Dario released a blog last week where he basically said these are extraordinary circumstances and the formal policy is too slow.
I took major issue with that because basically he's saying the government is too slow to respond to the fire that we started. So, therefore we have to suspend normal procedure. That is a playbook or a page right out of the playbook of what Hitler literally used to do to get control of Germany with the Reichstag fire. Now, I'm not saying that Dario is Hitler, but I'm saying that he's reverse engineering public manipulation from first principles.
I might slow down my blogs and video production because I'm really focusing on the audiobook now for Labor Zero. And it is a lot of work. Since I'm an amateur, I'm a little bit slower at recording the narration. But I'm practicing and I'm getting faster. Basically, my voice has only so many hours per day. So, I try and dedicate as much of it to practicing narration and actually recording narration.
Keep David Shapiro in your library
Save the videos and channels worth coming back to, and find them again in one place.
