In a sobering address delivered at the Royal Society in London, Dario Amodei, co-founder of the frontier AI lab Anthropic, has issued a stark ultimatum to the global technology community: artificial intelligence development must proceed only under rigorous human supervision. His warning comes as the industry barrels toward the next generation of systems, with capabilities that remain poorly understood even by their creators.
Amodei, a former OpenAI researcher and prominent voice in AI safety, argued that without a binding oversight framework, we risk creating technologies that evolve beyond our ability to control. “We are building entities that could, in short order, possess cognitive abilities far exceeding our own,” he told the audience. “To proceed without safeguards is not innovation; it is irresponsibility on a civilisational scale.”
The speech, titled “Handing Over the Reins: Why AI Cannot Be Left to Run Alone”, was part of a broader discussion on digital sovereignty and the societal impact of artificial intelligence. Amodei emphasised that the problem is not merely one of alignment – teaching AI systems to follow human instructions – but of governance. He called for an international treaty akin to the Non-Proliferation Treaty for nuclear weapons, but applied to advanced AI development.
Anthropic, the company he leads alongside Dario Amodei, has positioned itself as a safety-first alternative to rivals like OpenAI, prioritising interpretability and value alignment. Amodei warned that current approaches are insufficient and that the pace of progress is outstripping our ability to understand what these systems are doing. “We cannot audit a neural network of trillions of parameters the way we audit a corporate account,” he said. “We need new tools, new institutions, and a new culture of transparency.”
His remarks come at a time when regulators worldwide are scrambling to catch up with the speed of AI deployment. The European Union’s AI Act is the most comprehensive legal framework to date, but Amodei argued that it remains toothless and reactive. “The law is written for incremental technologies. AI is exponential,” he noted.
Critics of Amodei’s position argue that excessive caution could cede the frontier to authoritarian regimes, but he dismissed the notion that safety and progress are in opposition. “The safest path is also the most sustainable,” he insisted. “If we build without oversight, the inevitable accidents will lead to a public backlash that halts progress entirely. That would be the worst outcome for everyone.”
Amodei’s vision of a human-centred approach is not sentimental but practical. He proposed a tiered licensing system for AI systems above a certain capability threshold, with mandatory independent audits and shutdown capabilities embedded in hardware. He also called for increased funding for interpretability research, the field that aims to reverse-engineer neural networks to understand their decision-making processes.
The speech was met with a divided response from the tech sector. Some executives privately agreed but expressed concern about slowing down deployment. Others, like Jensen Huang of Nvidia, have openly argued that the benefits of rapid AI advancement – particularly in medicine and climate science – outweigh the risks. Amodei countered that this false dichotomy ignores the existence of catastrophic risks: “A cure for cancer is worthless if the infrastructure it runs on has been subverted by an unaligned intelligence.”
As the lecture concluded, Amodei struck a note of cautious optimism. “We have the technology to build a world of abundance,” he said. “But only if we have the wisdom to build it with our eyes open. The future is not somewhere we arrive. It is something we build together, with oversight, accountability, and a commitment to human dignity.”
The event was livestreamed, and has already garnered millions of views online, signalling the deep public concern over the rapid march of artificial intelligence. Whether policymakers heed his warning remains to be seen. But for now, a leading voice in the field has drawn a line in the silicon sand.
