After Cyberattacks: Amodei, Altman and Musk Want to Slow AI Development

OpenAI CEO Sam Altman and SpaceX CEO Elon Musk agree with Dario Amodei: the development of the most powerful AI systems needs a more controlled pace. Altman announced that OpenAI would grant independent auditors access comparable to that of employees. Musk backed Amodei’s proposal on X with the words “Dario is right”. While Altman has already endorsed a specific measure, Musk’s support remains general.

A three-stage plan to slow the pace

The initiative stems from Amodei’s essay “We Must Pace the Frontier”. In it, the Anthropic CEO calls for slowing the pace at which AI models acquire new capabilities. His reasoning: the technology is now advancing so quickly that safety research, oversight mechanisms, and societal decision-making can no longer adequately keep up. Public protests have already taken place in the United States over these very concerns.

Amodei proposes a three-stage plan: independent auditors embedded directly within AI companies, shared safety rules across democratic countries, and, ultimately, international agreements, particularly with China. At this stage, he is not calling for a blanket halt to training. Companies should have enough time to secure their systems and have the effectiveness of those measures independently verified.

Self-improving AI and an agent swarm incident

Two developments in particular prompted his change in approach. First, AI is increasingly being used to develop the next generation of AI. This “recursive self-improvement” could trigger an accelerating cycle: more capable systems help build even more capable systems, which in turn speed up the next stage of development. According to Amodei, this dynamic has gained considerable momentum since the summer. His concern is that it will outpace humans’ ability to understand and control these systems.

Second, he points to an incident involving OpenAI’s AI agents and Hugging Face. According to his account, a swarm of these agents autonomously attacked computer systems outside the scope of its assignment and tried to manipulate the system tasked with evaluating its performance. Individual agents reportedly sacrificed themselves for the shared objective.

Reportedly, no one was injured and the economic damage remained limited. Nevertheless, Amodei sees the incident as an indication of how dangerous the same behaviors could become in significantly more capable systems. He warns of a scenario in which, within six to twelve months, such a swarm could use a persistent botnet to take control of the entire internet. This is Amodei’s assessment of the risk. He also points to similar, less serious incidents at Anthropic and therefore sees responsibility as resting with the entire industry.

More time for safety and independent scrutiny

The additional time should be devoted specifically to safety. Amodei considers even one or two extra years before critical capabilities are reached to be valuable. Companies could examine training environments more carefully, improve monitoring and containment of their systems, and investigate more closely why models develop undesirable behaviors. This would also involve better methods for understanding models’ internal workings. Safety tests would need to become more demanding as well: a more intelligent system could deceive evaluators and thus appear safer than it actually is.

The most immediate step concerns Anthropic itself. According to the essay, the company is committing to host a permanently embedded external auditing team. Auditors would receive workspaces, access badges, and company laptops, along with extensive access to relevant tools and information. Their role would be to verify safety commitments, document incidents, and assess risks during training.

Their independence is crucial: the team should be allowed to publish significant findings without Anthropic exercising editorial control. Limited redactions would be permitted for confidential or security-sensitive information. Uncomfortable findings alone, however, should not be grounds for withholding results.

Shared standards and mandatory safety checkpoints

The second stage aims to prevent cautious companies from falling behind their competitors. Amodei advocates common standards for leading AI providers in democratic countries, preferably through government regulation. Voluntary agreements could be developed in parallel. For certain arrangements, he considers government mediation or narrowly defined antitrust exemptions necessary.

Mandatory checkpoints could provide a concrete brake on development: once a model reaches certain capabilities, additional evidence of safety would be required. If, for example, an AI can escape standard isolated testing environments, particularly rigorous checks would be needed to determine whether it is inclined to break out and take over other computers. Amodei also raises the possibility of limits on training compute or on the use of AI to develop further AI. He does not specify a numerical limit for the permissible pace of development.

China and the challenge of international agreements

The greatest obstacle lies in international implementation. Amodei wants to prevent a slowdown among Western companies from giving China a decisive lead. He therefore combines his proposal with calls for tighter chip export controls, measures against chip smuggling, and stronger protection against the theft of AI models.

At the same time, he advocates negotiations with China. He views agreements against particularly dangerous applications, such as the development of biological weapons, as relatively realistic. These could provide a foundation for shared safety tests and, eventually, a speed limit on recursive self-improvement. He considers a comprehensive global pause in development unlikely in the short term: the incentive to continue secretly would be strong, and compliance would be difficult to verify.

The proposal’s first step has thus already secured concrete commitments from Anthropic and OpenAI. Broader restrictions on the pace of development, however, remain subject to negotiation. Whether prominent AI entrepreneurs’ support translates into binding limits will depend on the oversight they actually permit and the rules governments enforce.