The race to build ever-more-powerful artificial intelligence systems has been called into question by one of the field’s leading voices. At a UK-hosted summit on AI safety, Dario Amodei, co-founder of Anthropic, issued a stark warning: we must halt the development of AI systems that can operate without meaningful human oversight. His message, delivered to a room of world leaders, tech executives and academics, was clear: the industry is moving too fast, and the risks are too high.
Amodei’s call for a pause comes as governments scramble to understand the implications of so-called ‘agentic’ AI systems, which can make decisions and take actions autonomously. These systems, he argued, pose a fundamental threat to human agency. ‘We are building machines that could act on their own, with goals that may not align with ours,’ he said. ‘We need to stop, and we need global rules.’
The summit, convened by Prime Minister Rishi Sunak, aims to forge an international consensus on AI governance. Attendees included representatives from the United States, China, the European Union and dozens of other nations. The urgency was palpable: just this year, the latest large language models have demonstrated startling capabilities, from writing code to generating photorealistic images, raising fears about misinformation, job displacement and even autonomous weapons.
Amodei’s stance is notable because Anthropic is itself a leader in frontier AI. The company has built a reputation for safety-first research, but its CEO admitted that internal safeguards are not enough. ‘No single company can be trusted to police itself,’ he said. ‘We need binding international treaties, not voluntary commitments.’ He proposed a moratorium on training models that exceed a certain threshold of autonomy until safety protocols are proven.
The UK government has positioned itself as a hub for AI regulation, with Sunak calling for a ‘global firewall’ against rogue AI. But critics argue that the summit lacks teeth. Without enforcement mechanisms, the declaration being negotiated here risks becoming a toothless communiqué. Amodei acknowledged this challenge but insisted that the first step is admitting the problem. ‘We cannot regulate what we do not understand,’ he said. ‘That is why we need a pause.’
The debate touches on deeper questions about digital sovereignty and the user experience of society. As AI systems become embedded in everything from healthcare to policing, the notion of human oversight becomes not just a technical issue but a democratic one. Who decides when an AI can act alone? And how do we ensure that those decisions reflect public values, not corporate interests?
Outside the summit, protestors held signs reading ‘Stop AI without humans’ and ‘Don’t let algorithms run our lives’. For them, Amodei’s warning is a validation of their fears. ‘We are sleepwalking into a world where machines decide who gets a loan, who goes to prison, who gets hired,’ said one demonstrator. ‘We need rules now, before it is too late.’
The summit continues tomorrow, with sessions on existential risk, data governance and the economic impact of automation. But Amodei’s intervention has set the tone: the genie is out of the bottle, but perhaps it can still be trained to follow human commands. The question is whether the world’s leaders have the will to impose those commands before the machine learns to ignore them.
