The race to build ever more powerful artificial intelligence must not become a contest run by machines alone. That was the stark warning from Anthropic co-founder Dario Amodei, speaking at a London technology summit this morning, as the United Kingdom moves to position itself at the vanguard of global AI governance.
Amodei, whose company is among the world’s most advanced AI labs, told a packed auditorium that the current trajectory of development risks creating systems that operate outside meaningful human control. “We cannot allow AI to develop without humans in the loop,” he said. “The stakes are existential. Not just for jobs or economies, but for the very fabric of society.”
His remarks came hours after the UK government unveiled a new regulatory framework for artificial intelligence, one that prioritises ethical oversight over breakneck commercialisation. The framework, which draws on recommendations from the AI Safety Institute, mandates that all high-risk AI systems deployed in the country must maintain a “human-in-the-loop” mechanism for critical decisions. This includes applications in healthcare, criminal justice, and financial services.
The timing is significant. Britain, once seen as a tech policy laggard, now finds itself at a crossroads between Silicon Valley’s “move fast and break things” ethos and Europe’s more precautionary approach. By backing Amodei’s vision, the government signals that it wants to carve a distinct third path: innovation tempered by accountability.
“The United Kingdom has a proud history of industrial leadership,” added Amodei. “But leadership in AI must mean more than publishing papers or building bigger models. It must mean building safe, trustworthy systems that enhance human agency, not erode it.”
Behind the scenes, the urgency is palpable. Anthropic’s own research suggests that frontier models could soon develop capabilities their creators do not fully understand. The company has championed “constitutional AI” as a way to hardwire safety into models, but Amodei admitted that no technical fix is sufficient alone. “We need regulators who understand the technology, who can ask the right questions, and who are not afraid to say no,” he said.
The UK’s Framework for AI Risk Management, published this week, attempts to do exactly that. It establishes a new oversight body, the Office for AI Safety, with powers to audit models before and after deployment. Companies that fail to comply risk fines of up to 10% of global turnover. The message is clear: the days of self-regulation are over.
Critics point out that enforcement remains untested. The AI industry moves at internet speed, while governments move at the pace of legislation. Whitehall’s digital teams are already stretched thin. Yet the mood in the room this morning was cautiously optimistic. For the first time, a major AI player was publicly calling for the very constraints that many in the industry privately resist.
Amodei did not mince words about the alternative. “If we get this wrong, we may not get a second chance. The technology will not wait for us to catch up.”
The London summit continues this afternoon with sessions on quantum computing and digital sovereignty. But the message from the keynote will linger: the future of AI is not a technical problem alone. It is a human one.
