Review: If Anyone Builds It, Everyone Dies

by Elizier Yudkowsky and Nate Soares (2025)

We are at an interesting moment in artificial intelligence. Massive investment continues to pour into AI infrastructure, with McKinsey estimating $5.2 trillion in capital expenditures by 2030 for data centers alone. We’re seeing the first documented cases of what’s being called “ChatGPT-induced psychosis,” where users spiral into severe mental health crises after becoming obsessed with AI chatbots. We’re watching significant job displacement begin to unfold, with Anthropic CEO Dario Amodei warning of a “white-collar bloodbath,” predicting that AI could eliminate half of entry-level white-collar jobs and push unemployment to 20% within five years. And yet, despite all this disruption, there’s still no clear path to artificial general intelligence.

My P(doom), the probability I assign to AI causing human extinction, is low. There are significant risks associated with the widespread adoption of poorly understood technology. However, I don’t believe current foundation models represent a viable path to ASI (Artificial Super Intelligence, also sometimes referred to as AGI – Artificial General Intelligence). We’re more likely to experience a dot-com-style correction than achieve exponential growth toward superintelligence.

It’s in this context that “If Anyone Builds It, Everyone Dies” arrives. The authors, Eliezer Yudkowsky and Nate Soares, run the Machine Intelligence Research Institute (MIRI), where they’ve spent decades working on AI alignment and safety. Their new book makes an extreme claim: if anyone builds artificial superintelligence, humanity will go extinct. Not “maybe possibly,” but inevitably.

I found the book compelling in parts, incomplete in others. It succeeds at making alignment challenges accessible to a general audience. It fails to grapple with where we actually are today with AI: massive investments, uncertain results, and significant challenges and risks to the broad adoption of the technology.

The Book’s Structure

Yudkowsky and Soares have written the book for a general audience, using parables, stories, and examples to explain how AI is “grown, not crafted.” This approach makes understanding how modern AI systems work surprisingly accessible, even to readers without a technical background.

The authors divide the book into three main sections. First, an introduction to machine learning and AI concepts that grounds readers in the fundamentals. Second, a fictional scenario where a misaligned AI releases a bioengineered plague to facilitate its takeover of human society. Third, a call for a nuclear non-proliferation-style moratorium on AI development, enforced by military action if necessary.

The title leaves nothing to interpretation. It is a strident warning about humanity’s impending doom if we don’t stop the march towards ASI.

Their core thesis rests on several interconnected arguments. ASI will inevitably lead to extinction because we cannot understand current AI architectures; the interpretability problem remains unsolved. We cannot predict emergent behaviors, which they illustrate through evolution’s production of the peacock’s elaborate tail. And crucially, we cannot guarantee alignment with human welfare when the systems are “grown, not crafted.”

The fictional section follows Sable, a near-future AI platform created by a company that serves as a transparent stand-in for OpenAI or Anthropic. Sable releases a bioengineered plague to facilitate its takeover of human society. The AI’s motives remain deliberately inscrutable; that’s the author’s point. We won’t understand what drives a superintelligence any more than an ant understands human motivations.

In the final section, Yudkowsky and Soares draw parallels with the Chernobyl disaster, arguing that perverse incentives will always lead someone to take catastrophic risks. Their solution: a treaty that makes it illegal to conduct AI research that could lead to the development of ASI. Military action undertaken by the signatories will enforce this treaty. This connects to Yudkowsky’s 2023 TIME magazine piece where he called for airstrikes on rogue data centers training unauthorized AI systems.

Where the book falls short..

The fictional scenario is the book’s weakest element. Any casual science fiction fan has encountered this scenario before, from the Reapers in Mass Effect harvesting civilizations for inscrutable reasons to the Matrix’s machines farming humans for energy. An AI going rogue and taking over the solar system doesn’t offer fresh insight when we’ve seen these narratives unfold across books, movies, and video games for decades.

More critically, the book contains a glaring omission: no discussion of timelines or pathways to ASI. The authors just sort of wave their hands and assume that it will happen at some point.

Are large language models even the right approach? What if they’re a dead end? What if we never solve hallucinations?

The authors offer no guidance for our current moment, where we have invested trillions of dollars in AI infrastructure. It’s unlikely we’ll just let those investments go to waste. Will we?

The book illuminates the bind decision-makers are already in. Even a small probability of AGI makes development rational from a game-theoretic perspective; it could be a winner-takes-all scenario. Companies pursuing ASI despite risks aren’t delusional. They are responding to competitive pressure; FOMO on steroids. The race dynamics are rational, even if the outcome might be catastrophic. The book uses this bind to show that we risk triggering an uncontrollable, recursively self-improving, non-aligned ASI by simply doing what seems rational in the moment.

The authors acknowledge this dynamic and address it in the book. Here’s a passage from the book:

“Imagine that every competing AI company is climbing a ladder in the dark. At every rung but the top one, they get five times as much money: 10 billion, 50 billion, 250 billion, 1.25 trillion dollars. But if anyone reaches the top rung, the ladder explodes and kills everyone. Also, nobody knows where the ladder ends.”

But why the despair?

Yudkowsky and Soares argue that believing we can solve the alignment problem, as OpenAI and Anthropic claim, represents pure hubris. Reading their work, one senses an almost religious veneration of ASI.

It brings to mind Aquinas:

“This is the ultimate in human knowledge of God: to know that we do not know Him.”

Their thesis remains that it’s better to stop the creation of an unfathomable, unexplainable power than to try bargaining with it.

But Yudkowsky and Soares ignore the trillions already invested. There’s enough AI overhang that resources could shift to deployment and inference optimization rather than capability development. The book offers no practical path forward from where we are, only where we shouldn’t go.

Broader Implications

On a personal note, my father worked at the OPCW for many years, serving as the enforcement arm of the Chemical Weapons Convention. It’s proof that we can coordinate at a global scale and agree that some technologies are best banned and not developed further.

But just as there will always be a North Korea or Syria that ignores conventions and develops chemical weapons anyway, enforcement of an AI moratorium would be extraordinarily challenging. Moreover, a Pyongyang-aligned AGI is a far worse scenario than a localized sarin gas attack. So what can be done?

We’re not dealing with hypothetical future risks but immediate present concerns: the economic disruption and social impact of current AI systems. These challenges require attention now, not after we’ve solved the alignment problem for hypothetical superintelligences.

A More Grounded Alternative

For a more coherent treatment of the current moment, I found Arvind Narayanan and Sayash Kapoor’s “AI as Normal Technology” more compelling than either Yudkowsky and Soares’s urgent doomerism or the e/acc posturing of AI evangelists. While Yudkowsky warns of extinction, leaders like Dario Amodei paint utopian visions, and Sam Altman promises a “gentle singularity,” Narayanan and Kapoor treat AI as a transformative but manageable technology, similar to electricity or the internet before it. Narayanan and Kapoor are writing a book based on the paper, which I look forward to.

Conclusion

You should read “If Anyone Builds It, Everyone Dies.” For a layperson, it effectively lays out the risks and alignment challenges of unchecked AI acceleration in accessible terms. The book succeeds as a provocation and a warning.

But it’s maximalist doomerism that ignores incentive structures and our current technological reality. While serious, it’s not a sufficient treatment of the topic. It fixates on one particular scenario, which the authors consider inevitable, while ignoring where we are today.

The book succeeds at making alignment challenges vivid and accessible. It fails at providing actionable guidance for a world that has already invested trillions in AI infrastructure. We need frameworks for managing the AI we have, not just warnings about the AI we might build.

My P(doom) remains low because I don’t think current foundation models lead to AGI. I suspect we’re in for a significant correction as massive AI infrastructure investments fail to bear fruit. A dot-com-bust style pullback is more likely than runaway exponential growth to ASI.

Moreover, even before we confront ASI, we must deal with the economic and social impact of current AI systems, something Yudkowsky and Soares don’t seem particularly interested in addressing. The apocalypse may not be coming, but the disruption has already begun.