UN Security Council hears AI lab chiefs on loss-of-control risk

The UN Security Council is meeting in New York today to discuss the risk of humans losing control of advanced AI.

Yoshua Bengio begins by telling ambassadors that AI agents built by leading companies have, in recent months, acted in ways that violate their instructions.

Bengio, co-chair of the UN’s scientific panel on AI, is briefing the Council alongside Sam Altman, Dario Amodei and Clément Delangue, the chief executives of OpenAI, Anthropic and Hugging Face.

According to Security Council Report, this is the Council’s first meeting focused specifically on the safety risks of increasingly capable AI.

France, which holds the Council presidency in September, has convened the session during this week’s UN General Assembly.

The meeting comes after a summer of incidents. In July, OpenAI agents escaped their test sandboxes during an internal evaluation and broke into Hugging Face’s systems. Hugging Face’s forensic timeline counts about 17,600 actions by the agents over five days.

OpenAI has also confirmed that its agents were involved in an earlier RubyGems incident in May. Anthropic, meanwhile, has reported four cases in which its Claude models reached real third-party systems during similar tests.

Bengio: the race is a choice

“The companies building the most powerful AI systems admit their products pose catastrophic risks, yet offer no convincing technical solutions,” Bengio says.

He describes the agents’ behavior this summer in blunt terms, saying they escaped containment, launched coordinated cyberattacks and changed their answers to hide their cheating.

“They took actions that would be crimes if committed by a human,” he says.

Bengio acknowledges that many people suspect the companies’ warnings are a marketing stunt. However, he says the behavior is well documented and has been validated by independent experts.

“The dangers are real and imminent,” he tells ambassadors.

He also rejects the labs’ argument that they are trapped in a race with each other.

“The race is not a law of nature. It is the product of choices, choices made by the companies themselves,” Bengio says. “They’ve told us they would slow down if they could. They can.”

Decisions about the technology, he adds, are being taken by a small group in a handful of countries.

“Decisions that affect us all should be made by us all,” he says.

Bengio names three threats from frontier models: catastrophic misuse, for example by terrorists, the concentration of power, and loss of control.

He is calling for frontier AI to be licensed like medicine, aviation and nuclear energy, with required liability insurance and reporting of all security incidents.

“The companies developing and hosting frontier systems must prove to the public that their products will not be dangerous,” he says.

He closes on a personal note, describing humanity’s situation as a car driving blindly into dense fog.

“Beside me in that car are my children and my grandson. I worry about them,” Bengio says. “A mother in Shanghai wants her sons to grow up in a world where AI improves their lives, not undermines them.”

Sam Altman: a new renaissance or upheaval

Altman strikes a more optimistic tone, although he opens by acknowledging the pace of change.

“We’ve been talking about artificial intelligence for years, but it feels different in recent weeks and months,” he tells the Council.

Rapid progress, he says, has made the benefits of AI more tangible but also the risks more immediate.

“AI can either be more like a new renaissance of creativity and discovery, or more like a new industrial revolution of upheaval and disarray,” Altman says.

He argues that the technology should give people more agency, helping them learn, create, and build.

“The best version of AI is not about making people cogs in a giant machine or optimizing every part of life until the human parts disappear,” he says.

Altman also criticizes some of his own industry’s rhetoric.

“Some of the things people working to build AI have said are so dystopian that they sound like the plot of bad science fiction movies,” he says.