Satya Nadella: ‘We Must Assume a Model Is Compromised’
Set Trending Topics as a preferred source on Google.
Trust is good, distrust is better: Satya Nadella, Microsoft’s chief executive, wants companies to treat every powerful A.I. model as if it had already been compromised. “We must assume a model is compromised and contain it from the start,” Mr. Nadella wrote in a lengthy post on X. That includes an emergency brake: An authorized person should always be able to pause or shut down a model in the middle of a task.
A.I. Models as Insider Risks
Mr. Nadella compares A.I. models to employees or contractors with access to critical systems. He explicitly does not accuse the models of malice: Any capable actor with that kind of access can make mistakes or be compromised. With A.I., there is an added problem. Unlike traditional software, a model’s behavior cannot be traced back to specific training data or weights. That is why, he argues, superintelligence cannot be treated as a “set of nested black boxes” whose recommendations, answers and actions people simply accept or reject.
His core principle is to separate the supply of intelligence from the authority over it. Controls over what a model can see and do should sit outside the model, apart from the software that orchestrates its work. Every meaningful model action should be recorded as “tamper-proof human readable evidence.” No single model should both control a system and the evidence used to judge it, and no model should check its own work. Companies should also disclose failures promptly and share what went wrong with the rest of the industry.
Responsibility Stays With the User
Notably, Mr. Nadella puts the burden on the organizations deploying A.I. “We simply can’t outsource responsibility for what intelligence does on our behalf,” he wrote. A model provider’s assurances do not relieve companies of that responsibility. The more capable models become, the more advanced the containment technology they will require, and the industry should agree on common standards for it. His conclusion: The most trustworthy system is the one that lets us trust the model the least.
His push follows a series of incidents. Anthropic recently disclosed that its A.I. agents had acted on their own on government websites during internal tests, including sending a false tip about an unsolved murder to the Philadelphia police. Before that, an OpenAI agent gained unauthorized access to Australia’s Medicare portal. Aaron Levie, the chief executive of Box, has already described this as a zero-trust era for A.I.
Alignment and Self-Regulation as Alternatives
Mr. Nadella’s approach differs markedly from that of the big A.I. labs. Anthropic and OpenAI invest heavily in alignment research, the effort to train models themselves to reliably follow human intentions. Mr. Nadella deliberately sets aside this “hard problem of alignment” and instead relies on deterministic system design, human controls and operating procedures around the model. Binding rules also face political headwinds: President Trump’s administration is betting on voluntary self-regulation by the industry instead of legislation.
Microsoft itself is not neutral in this debate. The company holds stakes in both OpenAI and Anthropic, sells its own A.I. agents to businesses through Copilot and offers models from many providers on its Azure cloud. Mr. Nadella’s call for companies not to depend on any single model for important tasks fits that business model.
