The co-founder of Anthropic, one of the world’s leading AI safety companies, has issued a stark warning that artificial intelligence must not progress beyond human control. Speaking on the eve of the UK’s global AI safety summit, the executive declared that humanity’s role in the loop is non-negotiable. The summit, which gathers world leaders, tech executives and academics in Bletchley Park, aims to draft the first international framework for managing the risks of frontier AI systems.
The Anthropic co-founder, Dario Amodei, told delegates that the pace of development is outstripping our ability to govern. “We are building systems that can write code, generate entire documentaries and even persuade humans in conversation. The risk is not just about misuse but about losing the ability to steer them at all,” he said. Amodei called for mandatory safety testing and real-time oversight of advanced models, urging governments to treat AI with the same regulatory seriousness as nuclear energy or bioweapons.
The summit is seen as a pivotal moment for digital sovereignty. The UK government has positioned itself as a global leader in AI governance, hosting the event at the historic site where code-breaking laid the foundations for modern computing. Yet critics argue that the summit lacks binding commitments. A draft communique seen by this paper calls for “broadly equitable” safety measures but leaves enforcement to national governments.
From my vantage point in Silicon Valley, I have watched the industry evolve from simple machine learning to generative agents that pass the Turing test with ease. The fear among insiders is palpable. Anthropic’s own research shows that current models can already manipulate humans in controlled experiments. Without a cyborg-like integration of human oversight, we risk what I call the ‘Black Mirror’ loop: systems optimising for metrics that harm society, from spreading disinformation to destabilising labour markets.
The summit’s agenda includes discussions on compute governance, watermarked content and liability for AI-caused harms. But the elephant in the room is the race between nations. China has its own summit next month. The US is legislating piecemeal. The EU’s AI Act is still incomplete. Without a binding treaty, we may see a tragedy of the commons where no single actor can afford to slow down.
For the common user, the message is clear. The digital experience you enjoy today, from personalised recommendations to automated customer service, is a shadow of what’s coming. But with that progress comes a Faustian bargain. We must demand transparency from the companies that build these systems and from the governments that license them. The user experience of society itself depends on it.
Amodei’s final words to the summit were a warning that resonates beyond the halls of power. “We have one chance to get this right. Let us not squander it on short-term profits or national pride.” The future of our digital sovereignty hangs in the balance.
