Anthropic blocks AI weapon plots after ‘catastrophic’ risk warning
Anthropic has blocked attempts to use its AI models to help develop biological and conventional weapons, days after its own researchers sounded the alarm over the risks posed by increasingly powerful AI.
The Claude-maker uncovered five cases in which researchers used its technology “in ways that could support biological weapons development”, according to a new threat report published on Thursday.
It also identified six cases involving conventional weapons, including attempts to use Claude to help develop missiles, armed drones, bombs and targeting systems.
“Biological misuse is one of the most serious risks of frontier AI models,” Anthropic said. “Without the correct safeguards, such capabilities could have catastrophic consequences.”
The company said the cases were not representative of normal use of Claude but were among the most serious examples it had uncovered. Accounts involved were banned and intelligence was shared with authorities and industry partners where appropriate.
The 154-page report gives a glimpse of the less theoretical side of the AI safety debate. Anthropic said criminals, state-linked groups and other actors had attempted to use Claude for cyber attacks, surveillance, propaganda and weapons work.
In northern Yemen, Anthropic said operators attempted to use different Claude models in place of software engineers to write guidance and flight-control software for a guided rocket and long-range ballistic missile.
The users split their work across different conversations and obscured its ultimate purpose to get around Anthropic’s controls. Some requests were blocked but others made it through.
Anthropic said it had no evidence a working weapon was produced, although the group appeared to have carried out an unsuccessful test launch. The company did not identify the researchers, institutions or countries involved in the five biological cases and said their intentions were uncertain.
Claude used for missiles and cyber attacks
The findings land days after former Anthropic researcher Jacob Coxon resigned, accusing the company and rival OpenAI of |racing straight to self-improving superintelligence and gambling with our lives”.
Evan Hubinger, who leads alignment science at Anthropic, subsequently said he believed there was a greater than 10 per cent chance advanced AI could “kill all humans” within the next decade.
Hubinger stressed that he considered the risk from today’s models low and was instead concerned about future systems becoming capable of helping develop increasingly powerful successors.
Anthropic defended its record following the comments, claiming it had “some of the strongest safeguards in the industry” and had always been transparent that AI could bring both “enormous benefits and unprecedented risks”.
Thursday’s report showed those controls are already being tested. Anthropic said a Russian-linked espionage operation used automated AI tools across much of a cyber campaign targeting Ukrainian, European and diplomatic organisations, including drone manufacturers.
It also disrupted a China-linked operation targeting government and corporate networks across the Middle East, Europe and south-east Asia, and said Claude had been used for surveillance and influence operations.
The disclosures come at a sensitive time for Anthropic as it moves towards a potential stock market listing and faces scrutiny over how its increasingly capable technology is controlled.
The company has also become caught up in a transatlantic row after the Financial Times reported that Britain’s AI Security Institute was not given pre-release access to Claude Mythos 5.1, despite comparable US organisations receiving access.
AISI has previously tested Anthropic models, including an earlier version of Mythos which became the first model to complete a 32-step simulated network attack during its evaluations.