AI researchers back an extinction warning as the Pentagon rejects it

A warning from a departing Anthropic researcher has drawn a wave of responses this week. Senior scientists at rival labs backed the claim that AI could threaten humanity, while a top Pentagon official rejected it. The reaction followed Jacob Coxon’s resignation from Anthropic. In a post on X, he said AI firms were “racing straight to self-improving superintelligence and gambling with our lives”.

The post has since passed 150 million views, according to Bloomberg.

Coxon said he had spent about three years in AI research across both OpenAI and Anthropic. He wrote that the people building AI earnestly believe it could kill everyone by the end of the decade. His posts reached more than 100 million people within days, according to Wired.

He told Axios he had left Anthropic two months before his equity would have vested, after roughly four months at the company. He no longer had anything to gain by juicing up Anthropic’s valuation, he said in that interview.

Colleagues and rivals back the claim

Several researchers at Anthropic and OpenAI publicly supported the warning. Evan Hubinger leads Anthropic’s alignment science team. He wrote on X that the company’s staff really do earnestly believe AI could kill all humans. He put the chance at more than 10 percent within the next decade.

Anthropic researcher Samuel Marks wrote that AI developers believe their technology could cause human extinction. In general, he said, the more senior the employee, the more concerned they were.

Paul Christiano also weighed in. He was formerly head of safety at the US Commerce Department’s Center for AI Standards and Innovation. Recent progress had led him to see a meaningful risk, he wrote. He said rapid acceleration in AI capabilities could lead to a catastrophic and irreversible loss of control in the very near term.

Most people could die, he added.

OpenAI announced that Christiano was joining the board of its nonprofit foundation. Geoffrey Hinton is one of the researchers often called a godfather of AI. He told BBC Newsnight that a 10 percent risk of human extinction seemed a not unreasonable estimate.

OpenAI researchers joined the calls to slow development. Julie Steele works on the company’s safety team. In her personal capacity, she wrote, she also thought the industry needed to slow down, CNBC reported. The statements followed an open letter in July. Roughly 1,400 employees across frontier AI companies signed it.

The letter urged the US government to build tools to deliberately pace automated AI development.

The Pentagon pushes back

Not everyone accepted the warning. Emil Michael, the US Defense Department’s chief technology officer, rejected it at a defence industry conference in Washington on Thursday, Bloomberg reported. It was easy to get caught up in this sort of doom loop, he said. He described Coxon’s message as all the fears that had coalesced in one well-written tweet.

Michael, a former Uber executive, said the free market, industry collaboration and engagement with government could mitigate the potential harms.

Michael also told reporters that the Defense Department had moved about 90 percent of its classified AI workload away from Anthropic. He said it was on track to shift the rest by the end of the month. His remarks followed the Pentagon’s dispute with Anthropic over safeguards on military use of its models, a case now before the courts.

Gary Marcus, an AI researcher, told Business Insider he saw cyberattacks and disinformation as serious risks. But he knew of no realistic scenario for actually killing all humans, he said.

The response reaches Washington

The warning drew a quick political reaction. Representative Lori Trahan, a Massachusetts Democrat, wrote that safety researchers were resigning, powerful AI models were breaking out of their labs, and companies were racing ahead anyway. She pointed to the bipartisan FRONTIER Act, which would establish oversight of advanced models.

A separate bill, the Ban Artificial Superintelligence Act, would pause advanced development until safety rules were set. Senator Bernie Sanders said he would introduce legislation to pause AI development. Senator Ted Cruz called the risk “scary as hell”, according to Politico.

David Sacks, chair of the President’s Council of Advisors on Science and Technology, addressed the matter on X. Anthropic’s planned public offering must be paused until the claims of the whistleblower can be investigated, he wrote.

Anthropic declined to comment on that post. The company is expected to begin marketing its initial public offering in mid-October at the earliest. The listing would follow shortly before the US midterm elections in November.

Anthropic’s position

Anthropic did not respond to a request for comment on Coxon’s resignation, and OpenAI did not immediately respond either, according to the Associated Press. An Anthropic spokesperson told CNBC that the company had always been transparent that AI would bring both enormous benefits and unprecedented risks.

The spokesperson said it was building models with some of the strongest safeguards in the industry. Anthropic was the first lab to publish a framework for mitigating catastrophic risks from AI models, the spokesperson added.

Coxon was not the first researcher to raise such concerns. Hinton left Google in 2023 to speak freely about the risks. In May, the Turing Award winner Yoshua Bengio issued a similar AI extinction warning.

OpenAI chief scientist Jakub Pachocki wrote this month that the systems of the next few years were likely to increasingly drive their own development, and that the moment called for extreme caution.

What set this week apart was the scale of the reaction, measured by view count and by the seniority of those who echoed the warning, rather than the argument itself.