Eliezer Yudkowsky – Big Yud to his enemies, of whom there are many – is one of the brightest lights in the alien constellation of Silicon Valley’s intelligentsia: the blaze of tech-adjacent druids, fascists, Buddhists, vegans, accelerationists and anarchocapitalists gleefully imagining the future the rest of us will have to live – or die – in.
Yudkowsky became internet-famous in the early 2010s when he published Harry Potter and the Methods of Rationality, a 2000-page fan-fiction serial in which the boy wizard is instructed in logic, cognitive psychology, game theory and rationalist thought. You can read this online and if you enjoy it you can donate money to the Machine Intelligence Research Institute in San Francisco, which was founded by Yudkowsky and promotes the argument animating much of his life’s work: that if anyone builds artificial superintelligence the human race will be destroyed.
This isn’t an original observation; the entire genre of science fiction has been sounding this warning for more than 100 years. But in the books and movies, the plucky humans generally pull through. Perhaps we short-circuit the computer by presenting it with a paradox; sometimes we teach it to love and cherish all life. Other times we blow it up.
For Yudkowsky and his co-author Nate Soares, the institute’s president, those scenarios radically underestimate the inhuman nature and likely capability of machine intelligence. It won’t reason in anything we recognise as language – some of today’s models operate more efficiently thinking in vectors of 16,384 numbers instead of words.
Transistors can switch billions of times a second; at best the neurons of our nervous systems spike a few hundred times a second. This means the machines already think orders of magnitude faster than we do – subjectively, a week for us is a century for them. A superintelligence will outperform all humans in all knowledge domains and cognitive tasks and multitask by spinning up new instances of itself by the million.
For Yudkowsky and Soares, there’s no war between humans and such a being. We just die. Perhaps it will be when the superintelligence turns all of the organic matter on the planet into computational resources or when it boils the oceans dry by using them for coolant. Or certainly after it has built a Dyson swarm – a megastructure surrounding the sun to capture all of its energy – and the Earth falls into darkness.
All of which sounds like science fiction. Yudkowsky knows people say this – he’s been arguing about the topic for decades, has heard every objection. Imagine jumping back 200 years and describing our world to your ancestors. Sounds like science fiction, doesn’t it? Now look at the rate of human technological change since the early 19th century, project it forward another 200 years and remember that two weeks for us is like two centuries for the superintelligence. It should sound like science fiction. Indeed, the reality will be weirder than the weirdest science fiction because our stories are constrained by the limits of human cognition and the superintelligence operates outside that.
Are we doomed? Not if we solve the alignment problem. Every programmer knows their code does exactly what they’ve told it to do, not what they want it to do – and exactly what they’ve told it to do often leads to programs consuming all a computer’s resources until it crashes. The alignment problem is the challenge of ensuring advanced AI systems reliably pursue the goals we intend rather than the goals we’ve explicitly set. Programmers can rewrite their code but AIs aren’t programmed, they’re grown. They’re not legible or rewritable in the manner of traditional software. At present we can retrain them or turn them off because we’re smarter than they are.
But what happens when we aren’t? Yudkowsky wants a moratorium on advanced AI research, the closure of large GPU (interconnected computer) clusters and a ban on large training runs until the alignment problem is solved. In a recent New York Times op-ed, he argued that it’s morally justifiable to carry out air strikes on rogue data centres. This did not endear him to the frontier AI companies.
A moratorium is unlikely. The “magnificent seven” – Apple, Microsoft, Nvidia, Tesla, Amazon, Meta and Alphabet – have all bet heavily on AI. The chips Yudkowsky wants to ban are manufactured by Nvidia, the world’s most-valuable company. The future of the species might be at stake but no one’s going to write off that much shareholder value.
So it’s comforting to look to the arguments against this theory of AI doom: the technology might plateau, or market incentives might constrain it – the cost of these models scales very steeply. Safety research is inventing new tools and alignment methods all the time. And today’s models are statistical tools with no agency. It’s not obvious they’ll become monomaniacal beings with grandiose goals once they hit some cognitive threshold. If you ask GPT-5 to assess Yudkowsky’s ideas, it replies they rely on a series of ingenious thought experiments rather than empirical forecasts, allowing him to present a highly speculative outcome as inevitable.
But the most sci-fi-friendly critique is that when we look up at night we see stars. If Yudkowsky’s form of machine intelligence is such a probable outcome of technological progress we should see darkness, as the power of those suns is harnessed for computation by AIs developed by other species.
Yudkowsky would scorn all of this as motivated reasoning: people want him to be wrong and build their arguments from there.
Most AI researchers admit there’s some level of danger from superintelligence. A recent leak from OpenAI revealed Sam Altman wanted to build a bunker to hedge against the risk of advanced models going rogue. How many chances do you want to take with the future of our species? And as for GPT-5’s criticisms – well, it would say that, wouldn’t it?
