A look at AI doomsday scenarios that researchers say could put humanity at risk

For years, artificial intelligence researchers have warned of ways the technology could wipe out humanity: AI bots could design an unstoppable disease, unleash a nuclear war, or transform all of Earth into paper clip factories, as one thought experiment goes.

Debates inside the AI industry and beyond over the plausibility of these doomsday scenarios have heated up since several executives endorsed slowing the technology's development for safety reasons.

Despite warnings of the potential for humans to lose their grip on the technology, some suspect the worst-case scenarios described by the AI companies themselves have more to do with validating the importance of their work than reality.

“These arguments just don’t really hold water. I think they’re sci-fi and they’re enticing to a certain childish style of thinking and it’s very tempting for the frontier labs because it helps them recruit certain types of folks,” said Juan Andrés Guerrero-Saade, a researcher at cybersecurity firm SentinelOne and a member of OpenAI’s Frontier Risk Council.

Still, more people are beginning to worry about the risks of the technology that has dazzled the world with its astounding capabilities.

Jacob Coxon, the Anthropic researcher who resigned from the company over concerns that it and its competitors are not acting responsibly in AI development, said on X recently that he needs to “work on” communicating to people more effectively exactly how AI would end humanity. “Unfortunately it is very difficult to convey specific scenarios,” he wrote earlier this month.

Here are some of the more widely discussed doomsday scenarios.

Nuclear winter

Researchers have long speculated that AI systems could trigger a nuclear war by getting countries to fire warhead-tipped missiles at each other. The nuclear blasts, and ensuing lingering radioactive atmospheric fallout resulting in a “nuclear winter,” could devastate life on Earth.

In a report last year, RAND Corp. researchers concluded that “at present” it wouldn’t be possible for AI to cause the use of nuclear weapons “because of strict safeguards built into the command and control systems.” However, the report said this could change if AI models are integrated into the nuclear weapon “decision chain” or gain unauthorized access to these systems.

For many AI researchers, that’s where the danger lies — not in AI agents suddenly deciding to end humanity, but people, nations or groups using it to wipe out their adversaries.

Bioweapons

One of AI's greatest promises is that it could help humans develop treatments and cures for diseases.

The flip side of curing diseases is creating or advancing them to harm humanity. Earlier this month, Anthropic said it has blocked efforts by bad actors to use its AI for a host of malicious activities such as cyberattacks, surveillance, and research that could have led to biological weapons.

In one instance, Anthropic said its systems blocked a request for assistance from its Claude tool for a research proposal seeking to enhance mutations to make the mosquito-borne chikungunya virus progressively more harmful. While such research could be used to develop better vaccines and treatments, Anthropic said, “it could also be used to make the pathogen more dangerous.”

Along the same lines, the RAND report noted recent progress in AI development “has the potential to transform the design of synthetic pathogens.”

Outlining a bioweapon doomsday scenario, the report said novel pathogens could be created in a lab, delivered to target locations, and spread to infect large populations of people in multiple places. But it still envisioned humans — and not autonomous AI — as overseeing and carrying out such an attack.

Runaway AI

Some AI researchers fear a human extinction event could result from “misalignment” — that is, when AI models act without human authorization, coordinate with other models or evade oversight. It would develop not from malicious intent — AI is not human, nor is it good or evil, after all — but simply because humans might get in the way of the model accomplishing its goal.

Take the decades-old “paper clip maximizer” thought experiment by the Oxford philosopher Nick Bostrom, which states that if advanced AI was told to produce as many paper clips as possible, it could eventually transform all of Earth, and then increasing portions of space, into paper clip factories.

This is to illustrate how superintelligent AI — often referred to as artificial general intelligence, or AGI — could advance a goal humans may see as desirable but “which in fact turns out to be a false utopia, in which things essential to human flourishing have been irreversibly lost,” Bostrom wrote in 2003.