The race to build ever more powerful artificial intelligence is accelerating without a critical safeguard: meaningful human oversight. That was the stark warning from Dario Amodei, co-founder of the AI firm Anthropic, as world leaders gathered in London for the UK’s inaugural global AI safety summit. Amodei, a former OpenAI researcher, cautioned that the current trajectory of AI development risks creating systems that operate beyond our comprehension and control.
“We are building machines that can make autonomous decisions with real-world consequences, yet we have no reliable mechanism to ensure they align with human values,” he told delegates. The summit, convened by Prime Minister Rishi Sunak, aims to forge international consensus on regulating frontier AI models. But critics argue that the industry-led approach risks becoming a talking shop rather than a binding framework.
Amodei’s intervention sharpens the debate. His company, Anthropic, has championed “constitutional AI” that hardwires ethical principles into models. Yet even this approach, he admitted, is insufficient without external oversight.
“We need independent auditors, not just internal safety teams. We need governments to define red lines for autonomous decision-making, especially in critical domains like healthcare, finance, and national security,” he said. The timing is poignant.
Just weeks ago, the UK government published a policy paper acknowledging that AI could pose “catastrophic risks” by 2030. Meanwhile, the European Union is finalising the world’s first comprehensive AI law, which classifies applications by risk level. The US has issued a voluntary code of conduct.
But Amodei argues that voluntary measures are a race to the bottom: “If one company cuts corners on safety to ship faster, it forces others to follow. We need mandatory standards, enforceable by law.” The summit’s agenda includes discussions on watermarking AI-generated content, preventing algorithmic bias, and ensuring that AI benefits are broadly shared.
However, a vocal group of civil society organisations staged a protest outside the venue, calling for a moratorium on new AI development until safety frameworks are in place. “We are sleepwalking into a world where autonomous systems make life-and-death decisions without transparency,” said Dr. Eleanor Green, a tech ethicist at the University of Oxford.
“The UK has a historic opportunity to lead, but it requires courage to challenge Big Tech’s narrative that speed trumps safety.” Anthropic’s own models, like Claude, are designed with “harmlessness” as a core objective. Yet Amodei admitted that even the best safeguards can be brittle.
“Every time we patch a vulnerability, we discover three new ones. The attack surface is growing exponentially,” he said. He urged delegates to focus on three priorities: mandatory stress-testing for high-risk AI applications, global standards for transparency, and investment in “interruptibility”: the ability for humans to override AI systems in real-time.
The summit runs for two days, with closed-door sessions between tech executives, policymakers, and academics. A final communiqué is expected to outline voluntary commitments. But for Amodei, the real test is whether those commitments translate into binding regulation.
“The future is not a predetermined script,” he said. “We write it with every decision we make today. Let’s ensure the story has a happy ending, for all of humanity.
