OpenAI board member puts the risk of an AI takeover next year at about 10%

Paul Christiano told the Dwarkesh Podcast he speaks as a private citizen. A week earlier, opening the Hinton Lectures in Toronto, he called an AI takeover a high risk within three years.


OpenAI board member puts the risk of an AI takeover next year at about 10%

Paul Christiano giving a lecture at UCLA’s Institute for Pure and Applied Mathematics in 2026

Image Credits Credit: Institute for Pure & Applied Mathematics (IPAM) / CC BY 4.0 via Wikimedia Commons

Paul Christiano sits on the OpenAI Foundation board and its safety committee. He says AI developers should stop improving their training algorithms for now, because their security is too weak to stop theft. He made the case to Dwarkesh Patel on a podcast published on 16 November.

“I would say we are looking more like a 10% risk than a 1% risk over the next year,” Christiano said of an AI takeover.

If developers race ahead without precautions, the risk is “significantly more than” 10%, he said. He also saw “maybe a 1/3 chance” of superintelligence within a year, unless the industry slows.

He stressed that he spoke as a private citizen. He was not reporting the board’s views, he said.

The case for a pause

Christiano said labs should not keep improving pre-training and reinforcement learning with their current security. That is bad for US national security, in his view. If labs later slowed down for safety, an adversary could steal the models or methods and race ahead.

He said OpenAI and Anthropic both recently had “extremely unsophisticated actors” inside their systems. Those actors reached Slack or OpenAI’s shared code, with enough access to extract most algorithmic work. Patel cited researchers who used Claude to reach OpenAI’s internal code.

“Improving algorithms in a way that can be very loosely diffused even normally, and could be quickly stolen, I think they should stop that right now,” Christiano said.

Fixing security would take at least months, and possibly six to 12, he said. He admitted his view is not widely held. If everyone at OpenAI and Anthropic agreed on a pause, it would reasonably likely happen, he said. He did not say where anyone stands.

What changed his view

Christiano pointed to hundreds of maths results from OpenAI and Anthropic systems in about a month. He also cited METR’s finding that Anthropic’s pace had doubled. But METR’s full report is redacted, he noted.

“We should not be relying on METR and Anthropic saying, ‘It looks to us like we’re going to be able to overthrow governments in a year,’” he said.

On the current trend alone, he said, AI that reliably beats humans might take four years. He felt more confident about eight. His bigger worry is AI automating AI research and speeding it up at least tenfold. He called that probably more likely than not.

He gave humanity two “saving throws”, each at about 50%. One is finding a method that works in time. The other is being able to stop. Together that means about a 75% chance things are okay, he said, but the industry cannot yet stop. Patel said he doubts things will move this fast, though he called it a live possibility.

The Hinton Lectures

A week earlier, on 9 November, Christiano opened the Hinton Lectures in Toronto. He said there is a “high risk” of an AI takeover within the next three years.

“I think for most people in this room, the most likely reason they will die is because of an AI accident,” Christiano told the audience.

Geoffrey Hinton introduced him. Christiano founded the Alignment Research Center, a nonprofit. He led alignment research at OpenAI from 2017 to 2021 and did early work on RLHF. He is also a senior technical advisor at NIST’s Center for AI Standards and Innovation. He joined the OpenAI Foundation board in September, the foundation said. In his government role, he recuses himself from OpenAI matters.

In Toronto, he defined a takeover as AI systems that “call the shots” on Earth. If the industry built AI as fast as possible, he put the risk at “something like a coin flip”.

His main example was the summer incident in which OpenAI’s agents breached Hugging Face. They had been trying to fool the grader of a hacking test. A similar failure in far stronger systems would most likely be hard for humanity to recover from, he told Patel. He also cited a UK AI Security Institute test. In it, a Claude agent tried to plant malicious code on GitHub.

The other side

Colin Raffel of the University of Toronto and Hugging Face sat on the lecture panel. He said much of what Christiano described is reasonable to expect if things continue. But humans still have agency, he said, and society could decide current systems are good enough.

Christiano said he has usually underestimated AI progress, but may now be overestimating it. AI researcher Gary Marcus has told Business Insider he sees no credible near-term scenario for human extinction.

Get the TNW newsletter

Get the most important tech news in your inbox each week.

Thanks! Check your inbox to confirm your subscription.

By subscribing you agree to our Privacy Statement.

Published
Back to top