Satya Nadella says he was wrong to think AI alignment would come with scale

Microsoft chief executive Satya Nadella no longer believes AI alignment will come just from making models more capable. He said so on a new episode of the 20VC podcast. Host Harry Stebbings had asked what he had changed his mind about in the past year.

“I used to always think that alignment will just fall out. I no longer believe that. So I think this alignment thing may be a harder, different problem,” Nadella said.

Nadella said he believed in scaling laws in 2018 and 2019. Back then, Dario Amodei, Ilya Sutskever, Sam Altman, Greg Brockman and Elon Musk were “one happy family”, he said. Scaling, the idea that models improve with more data and computing power, has worked as he expected. Alignment, making sure AI acts in line with human values and goals, has not come with it, he said.

Alignment starts in training

Nadella said alignment mostly has to happen during training. It may even need to start in pretraining, the first and largest stage of building a model. He also said the science of how models work inside remains elusive.

“I don’t think you can even RL your way to alignment,” he said.

RL, or reinforcement learning, is the method labs use to refine models after pretraining.

A zero trust system around the model

In the meantime, Nadella wants companies to build security around the models. He called for zero trust, an approach that checks every action instead of trusting anything by default.

“If you really want to have agents that are more capable and yet less aligned, then you need to have a trust system that you can rely on,” he said.

Microsoft’s Agent 365 and Perception give every agent an identity and log its tool and model calls, Nadella said. The logs must be tamper-proof, so an agent cannot change its own record. Companies then set policies and watch whether agents follow them. It added tools to monitor agents earlier this month. On 10 October, Nadella wrote on X that companies should treat AI models as insider risks.

Chain of thought and the labs

Nadella also said nobody can rely on the chain of thought. That is the step-by-step reasoning a model shows before it answers.

“We know that the chain of thought itself can be unfaithful, and in fact it can be made worse when the model company says, ‘Well, it’s unfaithful and I’m going to hide it,’” Nadella said.

He did not name a company. He wants trusted partners to get the chain of thought in plain English. They could then audit it and check it with other models. If makers of closed models do not see the need for more transparency, “we may have a problem”, he said. That could push more customers to open-weight models, whose weights anyone can download and run.

Open models and China

Stebbings pointed out that most open-weight models are Chinese. Nadella said that AI without safety is bad for China, the US and the UK alike.

“An unaligned open-weight Chinese model will probably attack even China,” he said.

Microsoft will not release its own MAI models as open-weight, Nadella said. He described them as licensed weights, sold as a straightforward software business. He expects most AI use to run on cheap commodity tokens from many competing models, open and closed.

Microsoft reports quarterly earnings on Wednesday, Todd Bishop reported for GeekWire. They are its first results since it offered buyouts to about 60,000 staff.