In a stark warning delivered to policymakers in London, the co-founder of Anthropic, the AI safety company behind Claude, has urged the UK government to halt the deployment of autonomous AI systems that operate without meaningful human oversight. Speaking at a closed-door session of the Science, Innovation and Technology Committee, Dario Amodei cautioned that the rapid advancement of large language models and agentic AI could erode human agency and, ultimately, democratic control over technology.
"We are sleepwalking into a future where algorithms make decisions that affect our lives, our jobs, even our justice system, without a human in the loop," Amodei said. "Britain has a choice: embed human values into AI now, or lose the ability to govern your own technological destiny."
The warning comes as the UK government advances its AI Safety Summit agenda, positioning Britain as a global hub for AI regulation. But critics argue that the current pace of deployment outstrips safety research. Amodei pointed to examples of AI systems already writing code, negotiating contracts, and even generating political propaganda with minimal human intervention. "The black box is becoming a black mirror," he added, referencing the dystopian series that explores the unintended consequences of technology.
The call for a moratorium on fully autonomous AI echoes similar demands from other tech leaders. In March, an open letter from the Future of Life Institute, signed by Elon Musk and Apple co-founder Steve Wozniak, called for a six-month pause on training AI systems more powerful than GPT-4. However, that appeal was largely ignored. Amodei believes Britain can set a different precedent. "You don't need to stop innovation. You need to stop recklessness."
Central to his argument is the concept of "digital sovereignty": the idea that nations must retain control over their digital infrastructure and the AI systems that run on it. Without such control, he warned, Britain risks outsourcing its decision-making to unaccountable corporations or even hostile state actors. "If your AI system is making life-or-death decisions in healthcare or criminal justice, who is accountable when it errs? The algorithm? The developer? The government that deployed it?"
Amodei proposed a three-pronged approach: mandatory human oversight for any AI system that makes high-stakes decisions; transparency requirements for training data and model behaviour; and independent audits that can pierce the 'black box' of deep learning. "Regulation is not the enemy of innovation. It is the scaffolding that allows innovation to scale safely."
The reaction from Westminster was mixed. Some MPs welcomed the focus on safety, while others worried about hamstringing a sector that attracted over £5 billion in private investment last year. Labour MP Chi Onwurah, former chair of the Science and Technology Committee, cautioned against panic. "We must not let the perfect be the enemy of the good. AI brings huge benefits: faster drug discovery, personalised education, efficient energy grids. But Amodei is right that we need guardrails."
The debate will intensify next month when the UK hosts the Global AI Safety Summit, where the question of human oversight will be high on the agenda. For his part, Amodei remains cautious but hopeful. "Britain has a window, a brief one, to shape the future. Close the window and you'll be looking through a screen you no longer control."
