By now, you’ve no doubt heard of Jacob Coxon. He’s the guy who resigned from his job at Anthropic in a viral tweet and now seems to be haunting the internet with his presence. Jacob worked at both Anthropic and OpenAI and suggests we are on a collision course with human extinction, and soon. The claim is absolutely preposterous, but keep in mind, he’s huffing the fumes from the AI psychosis factory. So, anything goes at this point.
I really didn’t want to write about this and have other topics I’m exploring, but it seems this story, as well as this topic, isn’t dying. Thankfully, many have written thoughtfully on this topic in recent days. One is Gary Marcus, and you can read his write-up here. In this article, I’ll address some of Jacob’s statements more directly and explain why people get sucked into the AI extinction narrative in the first place.
Human Extinction
Human extinction at the hands of a superintelligent AI is the doom scenario you wish would go extinct, but each time you think it’s dead, it emerges again like a phoenix rising from a sewer.
Here’s the start of Jacob’s tweet thread.
He has also stated there’s a strong chance we will all die in the immediate future. It’s odd because his high confidence relies on completely speculative technology that doesn’t exist. It assumes a fast takeoff scenario based on recursive self-improvement, something we don’t have evidence for today. And artificial superintelligence, which is speculative. Although why let a little reality get in the way of your doom vibes?
The extinction narrative surrounding AI won’t die for a couple of reasons. One reason is that people love science fiction tales, especially dystopias. People connect with these narratives because they’ve been exposed to so many fictional stories where things have gone wrong. I mean, if nothing goes wrong, then you don’t have a story.
Someone who watches movies about how robots want to kill all the humans gets the impression of not only the possibility but a higher likelihood of this scenario. This generates a false sense of understanding. Fiction often requires suspending disbelief, but the problem is that many never resume after the story ends.
People pushing doom narratives are often caught in bubbles. They are constantly fed more people with more doom narratives. Doomers also fall victim to audience capture, where they push more content to get more attention. Why else would you write a book called “If Anyone Builds It, Everyone Dies”? We’ll cover this a bit later in the article.
Another reason is that the AI labs love the attention from AI doom narratives. If AI is powerful enough to wipe out humanity, then that means AI is powerful. Powerful AI is profitable AI. Now investors give us more money!
You’d think AI companies would try to silence Jacob, or at least push back with a counter-narrative, but of course they aren’t. They are doubling down.
Obviously, the labs think the doom narrative is good for business. And this is a game they all have to play. If Sam Altman says that OpenAI could cause human extinction and Dario Amodei doesn’t follow suit, then it looks like OpenAI has the better models. It’s all incredibly stupid.
I’ve called this human extinction humble bragging in the past. Here’s an example from a 2024 post on AI performance art and absurd predictions.
Even when Altman and others talk about the potential of their technology to destroy humanity, it’s a sales pitch. They claim their technology is so good and so powerful it could wipe us all out, so please give us money. This is something I referred to before as the human extinction humble brag.
I’ll close this section with a graph from a survey by AI Impacts. See if you can spot the issue.
Seems to me, being disempowered is quite a bit different than being dead. If we are talking about disempowerment, then my p(doom) is certainly higher than 0.
The Door Never Closes On Destruction
In a report from RAND on the extinction risk from artificial intelligence they find no evidence that AI could cause an extinction event, yet they begin their conclusion with the following sentence.
Although we could not show in any of our scenarios that AI could definitely create an extinction threat to humanity, we could not rule out the possibility.
Nobody ever closes the door on destruction. Uncertainty creeps in and is stated in a way that makes it seem more likely. Sure, there’s always a possibility of incredibly improbable things happening or of scenarios we didn’t envision. Fair enough, but it should be stated as such. It’s these incredibly minuscule probabilities that doomers are latching onto and blowing out of proportion.
Details Matter
In the previous section, I included a tweet from Evan Hubinger who works at Anthropic. He stated that he personally thinks there is a greater than 10% chance that AI could kill all humans within the next decade. Not surprising, as Evan was formerly at MIRI (Eliezer Yudkowsky’s organization). But okay, how?
Whenever someone claims that AI will cause human extinction or that AI and nuclear weapons have the same destructive power, always ask that person how. Make them explain the scenario, because the details matter. You’ll often find it’s nothing but vibes all the way to destruction.
Doomers never have any specifics. It’s always some fuzzy, detail-free scenario that doesn’t account for the complexities of the real world because the common response is that a super-powerful AI will just “figure it out.” Sorry, that’s magic, not technology.
Take the paperclip maximizer. This example constantly comes up in these discussions. The scenario is supposed to demonstrate how a misaligned system can have negative consequences. It can, but people have started taking its global-scale impacts more literally. Sure, like no human on earth would be able to identify a paperclip factory running off the rails before it gets to human extinction.
Another one of my favorites is that AI will be so powerful it will manipulate people to their will and convince us all to kill ourselves. Proponents of this theory point out that AI is good at manipulating humans. However, what they leave out, and something I’ve pointed out many times, is that it’s also easy for humans to manipulate AIs.
Humans are bad at probabilities, and AIs are good at them. Both conditions open the door to manipulation. It’s relatively easy to make an AI system unsure of its decisions and reduce its confidence. After all, an AI system capable of causing human extinction needs to be connected to the real world, and once it is, it’s also vulnerable to the real world.
Humans are bad at probabilities, and AIs are good at them. Both conditions open the door to manipulation.
You typically find that all of these doom scenarios haven’t been thought through, and when you bring up these counterarguments, doomers will shrug them off because AI is AI.
When I have had conversations with people with a higher p(doom) than my own, they always push me to provide evidence, but that’s not how evidence works. If you tell me creatures like Bigfoot exist and I say they don’t, you can’t put the burden of proof on me to prove they don’t exist.
Inside The AI Psychosis Factory
We need to keep in mind that these people live and breathe in a factory for AI psychosis and AI-fueled delusions. I’ve previously written about how smart people get captured by AI hype. Now, imagine someone like Jacob, not only being firmly encased in a bubble, but also huffing the fumes of high-test hype directly from the source.
For example, here’s one of Jacob’s tweets from his public resignation:
In short, he’s saying that we are close to building a digital god. But this tweet demonstrates a stunning ignorance of reality. Let’s look at these claims.
Hacking Anything
In the wake of Project Glasswing and the OpenAI Hugging Face incident, the narrative is that AI can and will hack everything. This tracks with Dario Amodei’s recent post about swarms of agents taking over the internet in 6 to 12 months. The ridiculousness of which Gary Marcus, Zack Korman, and I addressed. That’s right, Dario says we are a mere 6 to 12 months away from making the dream of Zero Cool a reality.
Let’s address the “hack anything” narrative. In order to “hack” something, there first needs to be a vulnerability, something for AI to exploit. No vulnerability, no exploit. No hacking the planet.
Also, purely having an exploit for a vulnerability doesn’t necessarily mean it will be useful for an attack. The world and technical environments we encounter are complex. We can see this in the data: despite AI identifying more vulnerabilities, the vulnpocalyse hasn’t arrived.
However, we shouldn’t be complacent. In the real world, cybersecurity is far from solved. Many companies treat cybersecurity as an afterthought and invest far too little when they invest at all, creating a landscape of unpicked fruit. AKA vulnerabilities. And I should also point out that adding AI to applications and systems can make them even more vulnerable.
In my previous post on the Hugging Face incident, I called out the Gym Class Guy scenario, where his assistant found a “vulnerability” in the gym class booking application. Here’s what I said:
What AI highlights in scenarios like the gym class booking is the reality that many of us working in cybersecurity services have witnessed firsthand for decades. Most organizations have gotten away with lax application security processes and haven’t placed enough emphasis or allocated sufficient budget for cybersecurity. The gym class booking issue was a simple authorization bug, not a complex zero day. Any competent tester would have found this bug.
What the use and increased cybersecurity capabilities of AI mean for companies should be clear at this point. The era of companies getting away with leaving their low-hanging fruit unpicked is coming to an end.
In the end, Dario’s warning shouldn’t be about the power of AI but the continued lack of investment in securing applications and systems. Companies have gotten away with lax cybersecurity for decades, and that era is ending.
Another thing to consider is that a theoretical AI this powerful would also have defensive capabilities. So, defenders would have access to the same systems. Although an imbalance may exist between attacker and defender, these capabilities would still exist.
Revolutionize Any Field Overnight
Unlike the hack anything claim, which is an actual claim that can be addressed, the claim of revolutionizing any field overnight isn’t a claim at all. Any field? How? Using what methods? Using what data? Once again, this is complete nonsense that shows an ignorance of the real world and how things actually work.
If we look at something like drug discovery, it can take years to move from a breakthrough to an approved therapy. We can see this with advancements like AlphaFold. If we put the revolutionize any field overnight claim in context, it doesn’t account for conditions like data availability, regulatory approval, infrastructure, and the years of validation required to go from breakthrough to deployed change in a field. There’s a very wide gap between “AI can generate a hypothesis” and “AI can restructure an entire industry.” This is exactly the gap these claims never address.
Unfortunately, this isn’t an uncommon perspective from many working in tech. They know the tech, but they lack domain expertise in the areas where it’s applied. Domain expertise is key to solving problems in the domain where they exist, something that many fail to recognize as they refer more to magic than technology.
Acquire Real Power and Resources
What does this claim even mean? That the earth will be transformed into computronium? Once again, it’s not an actual claim that can be refuted. It’s a magic spell conjured to invoke fear.
If we are humans about to become extinct, you mean to tell me we wouldn’t notice an AI system acquiring power and resources? Acquiring power and resources on this scale requires physical infrastructure like compute, energy, factories, logistics, and even human labor. All of these are visible, monitored, and billed to someone. This would be the biggest industrial buildout in history. Seems like you’d notice the end of the world on your credit card bill long before the extinction event.
“Hmm, what are those factories building military equipment across the globe? Well, I guess the AI knows what it’s doing.”
Calling For A Pause
We are led to believe that because of this unprecedented threat to humanity, the AI labs themselves are now calling for a pause on AI development. Oh, I’m sorry, “pacing” AI development. Wink wink. They’re also calling for regulation. You might wonder why they are doing this and that it would be bad for business, but it’s business all the way down.
It’s no coincidence that other AI leaders are using the current moment and the AI safety narrative as an excuse to slow down and have regulation thrust upon them. Here’s Sam Altman saying the reason OpenAI won’t go public this year is because of AI safety issues. Yeah, sure. That’s totally believable.
A pause, pace, or whatever you want to call it would mean throwing less cash into the furnace. None of these companies are profitable, so burning less cash helps their bottom line. Especially as they try to go public. And, by having their competition follow suit, they ensure that they won’t be left behind, or at least that’s what they believe.
However, calling for regulation can also lead to regulatory capture. This regulation can be wielded to further shore up the business and stifle competition. So, although a call for a pause and regulation may seem like a concern and a good-faith move for humanity, it has nothing to do with it. It’s all about the business.
Here’s something to think about: If AI labs were close to achieving artificial general intelligence (AGI) or some fast takeoff scenario, they wouldn’t be calling for a slowdown.
If AI labs were close to achieving artificial general intelligence (AGI) or some fast takeoff scenario, they wouldn’t be calling for a slowdown.
Superincompetence, Not Superintelligence Will Kill People
Look, I don’t doubt for a moment that labs are acting irresponsibly, but to say that the consequences of this irresponsibility are human extinction is ridiculous. Could people die because of this irresponsibility? Certainly, and they no doubt have. But superincompetence, not superintelligence, is what kills people with AI.
The real danger to humans from AI isn’t the fact that the AI is superintelligent. It’s that a human decides to deploy it in a high-risk or safety-critical scenario, causing harm or death. Imagine hooking ChatGPT up to air traffic control or a GPT-powered autonomous weapons system.
The ways AI-powered human incompetence can be deployed to harm people are limitless, and this can happen without the need for artificial superintelligence. This is the real risk to humanity from AI. Ourselves.
This is the real risk to humanity from AI. Ourselves.
Conclusion
The whole human extinction at the hands of AI narrative does more harm than good and distracts from AI’s real risks. The AI doomer narrative can be destroyed with a couple of follow-up questions and requests for details. You won’t change the minds of the true believers, but you can reduce the blast radius to others.
In the end, I wouldn’t be surprised if Jacob is garnering all this attention so he can launch a new startup. On second thought, maybe we deserve extinction after all.