Anthropic researcher quits, warns AI could threaten humanity

Jacob Coxon, a senior researcher who spent three years on model training at both Anthropic and rival OpenAI, announced his resignation on Wednesday. He said the companies were "racing straight to self‑improving superintelligence and gambling with our lives" and were not acting responsibly as they push toward ever more powerful systems.

Resignation and safety alarm

The departure was quickly echoed by Evan Hubinger, who leads alignment science at Anthropic. He warned that the chance of AI wiping out humanity could exceed ten per cent within the next decade. "

Jacob is correct here, we really do earnestly believe AI could kill all humans,

" Hubinger wrote on X, adding that the firm lacks a clear plan to solve alignment for superintelligence.

Hubinger clarified that current models pose a low risk, but the prospect of recursive self‑improvement, where AI designs ever more capable successors, could change that dramatically.

Political and regulatory fallout

Former chief secretary to the Prime Minister Darren Jones responded by urging a multinational treaty to govern superintelligence development. In a letter addressed to Andy Burnham, he cited a joint note from António Guterres and Mathias Cormann urging governments to raise the issue at upcoming G7 and G20 meetings.

Jones argued that "the debate ranges from the end of humanity to claims of ‘marketing hype’ pre‑IPO" and called for a safety‑first approach without stifling innovation.

Implications for the UK AI ecosystem

At the same time, the UK’s access to Anthropic’s latest model, Claude Mythos 5.1, has been limited. The AI Security Institute (AISI) did not receive pre‑release access, according to a report by the Financial Times. Instead, a less‑restricted version was offered to a handful of vetted US organisations.

A spokesperson for the Cabinet Office defended Britain’s AI security posture, noting that the institute continues to collaborate with industry and recently tested OpenAI’s upcoming GPT‑6 Astra model.

Other internal voices have also spoken out. Samuel Marks, Anthropic’s scalable oversight lead, warned in a personal capacity that developers believe their technology could cause human extinction within years, driven by commercial pressure and competitive dynamics.

Company executives, including CEO Dario Amodei and co‑founder Jared Kaplan, have signed a petition with over a thousand AI workers calling for an international framework to slow frontier AI development. Yet Jones and other politicians are pressing for a formal treaty.

In a related development, policy adviser Matt Clifford stepped down from chairing the Advanced Research and Invention Agency after taking a full‑time role at Anthropic, following criticism over a potential conflict of interest.

With leading researchers publicly questioning the trajectory of AI and governments scrambling to craft oversight, the industry faces a pivotal moment. If regulators act swiftly, they could shape a safer development path; if not, the race to ever‑more capable systems may continue unchecked.