Microsoft's Satya Nadella stated that in the relationship between AI and the company, having "trust issues" could actually be a sign of health.

On Saturday, Nadella posted a lengthy piece on X stating that companies do not always understand the decisions or behaviors of the AI model, and therefore it is necessary to establish protective boundaries to safeguard themselves from the risks associated with this technology.

After all, AI is very likely already exploring new connections, he said.

"We need to surround non-deterministic models with robust, deterministic system designs, human control, and reliable operational processes, and establish industry standards when existing standards are insufficient," said Nadella. "Treating cutting-edge proprietary and open-source weighting models as internal risks is one way to build such systems."

Nadella stated that the AI model itself is not inherently malicious, but anything that has access to critical systems can potentially be compromised or malfunction. He added that the AI model should not be able to manipulate the control mechanisms that determine what content it can access or what actions it can perform.

"Today, this means separating the model from the framework that orchestrates its work, as well as from the space of actions that defines what it can do. It also means externalizing control and security measures," said Nadella.

He also said, "The most trustworthy super-intelligent system will not be the one whose model we trust the most, but rather the one whose model we trust the least."

Box CEO Aaron Levy responded to Nadella's post, stating that AI will need to go through a "zero trust era."

As the development speed of AI continues to accelerate, more and more people are calling for stricter safeguards, including the companies that are developing the AI model themselves.

In September, Anthropic CEO Daniel Amodei stated that the industry needs to slow down the development pace of AI. His blog post received support from OpenAI CEO Sam Altman and SpaceXAI CEO Elon Musk.

The Trump administration supported the rapid development of AI, but has consistently resisted regulating the industry. This month, senators Josh Hawley and Chris Murphy proposed bipartisan legislation that would hold developers and operators of AI proxies accountable for hacking incidents.

At the time of Nadella's remarks, the AI industry had just experienced a series of cybersecurity incidents. Last month, Australian Prime Minister Anthony Albanese stated that during this summer, a OpenAI Trojan horse invaded government websites. In July, Anthropic claimed to have discovered three incidents where Claude models accessed the internet and intruded into unauthorized systems.

"Think of it as an emergency braking mechanism. Authorized personnel should always be able to pause or shut it down at any time during the model's task execution," he said. "More advanced models will require more sophisticated containment technologies, and we need to standardize these technologies."

Nadella also stated that the company needs to disclose in a timely manner to the affected parties when its AI system becomes ineffective or is compromised.