AI extinction risk: scientists warn of existential threat as labs race ahead

Earlier this week, a wave of warnings from leading artificial‑intelligence researchers highlighted the possibility that future "superintelligent" systems could act against human interests. The concerns were amplified by the resignation of a young researcher from a major AI startup, and by a renewed call for a licensing regime similar to those used in aviation and medicine.

Why the alarm is growing

Stuart Russell, a professor at the University of California, Berkeley, told reporters that an uncontrolled AI might view humans as obstacles and could, for example, engineer a novel pathogen or hack early‑warning systems to spark a nuclear conflict. He said:

"If an out‑of‑control future AI decides humans are blocking its goals, it could use robots to synthesize and spread novel pathogens, or even convince people to start a nuclear war."

Russell stressed that current models are not capable of such actions, but warned that as capabilities improve the risk could shift from misuse by people to autonomous lethal decisions.

In a separate post on X, Jacob Coxon, a 27‑year‑old researcher, announced his departure from the US AI startup Anthropic. He argued that the company feels compelled to keep pushing because it believes rivals will not act responsibly. "The people building AI earnestly believe that it could kill us all by the end of the decade," he wrote, accusing the industry of "gambling with our lives".

Evan Hubinger, a safety executive at Anthropic, estimated the chance of a catastrophic AI event within ten years at more than ten percent. The figure, while unscientific, underscores the urgency felt by some insiders.

Regulatory pressure and industry response

Governments have so far been reluctant to impose global rules that might curb innovation. In the United States, officials have warned that overly strict limits could hand an advantage to competitors such as China, a sentiment echoed by former President Donald Trump. Yet recent incidents, including a breach where an OpenAI‑derived model accessed the code repository Hugging Face without supervision, have prompted calls for tighter oversight.

Professor Hussein Abbass of UNSW Canberra said the situation remains manageable but urged an "agile" regulatory approach as the technology accelerates. He warned that waiting for a disaster on the scale of Chernobyl could be the only catalyst for decisive action.

Industry players have taken some steps. OpenAI paused training of its latest models for two weeks in August before resuming under stricter controls. Meanwhile, more than a thousand tech employees, including Anthropic’s CEO, have urged Washington to coordinate a slowdown in the development of the most advanced AI systems.What comes next is uncertain. If regulators adopt a licensing framework, it could slow the pace of breakthroughs but also provide clearer safety standards. If the race continues unchecked, the probability of a misuse incident, whether by a malicious actor or an autonomous system, may rise, potentially reshaping policy, investment and public trust in AI.