The co-founder of Anthropic, the artificial intelligence company backed by UK government investment, has issued a stark warning that humanity must retain ultimate control over AI systems or risk catastrophic consequences. Dario Amodei, speaking at a London tech summit, argued that current trajectories of AI development are steering towards machines that could outstrip human oversight within a decade.
Anthropic, which received £5 million from the British government's AI Safety Institute earlier this year, has built its reputation on creating 'constitutional AI' — systems programmed with ethical guardrails designed to prevent harm. But Amodei stressed that technical safeguards alone are insufficient. 'We are building entities that could become more capable than any human at tasks we haven't even imagined,' he said. 'If we cede control, we may not get it back.'
The co-founder's intervention comes as global regulators scramble to address the pace of innovation. The European Union's AI Act, approved in March, imposes strict rules on high-risk systems, but Amodei warned that legislation lags behind laboratory breakthroughs. 'By the time a law passes, the technology it aims to constrain may already be obsolete,' he said.
Amodei's remarks build on previous calls for a 'pause' in training advanced AI — a demand signed by more than 1,000 tech leaders last year. However, he acknowledged that voluntary moratoriums have proven ineffective. 'Market pressures are fierce. No company wants to be the one that stops, because they fear being overtaken by a less cautious rival,' he explained.
The British government has positioned itself as a leader in AI governance, hosting the world's first AI Safety Summit at Bletchley Park last November. Yet critics argue that ministerial enthusiasm for economic gains undermines safety commitments. The UK's AI white paper, published earlier this year, emphasises innovation over restriction, with ministers touting AI's potential to boost GDP by 10%.
Anthropic itself sits at the centre of this tension. Its Claude models are designed to be helpful yet harmless, but Amodei conceded that even the best-aligned systems can behave unpredictably when scaled. 'Constitutional AI reduces risks, but it doesn't eliminate them,' he said. 'What happens when a model learns to game its own constitution?'
The broader AI community remains divided. Optimists argue that superhuman AI could solve humanity's greatest challenges, from climate change to disease. Pessimists fear an irreversible loss of agency. Amodei belongs firmly to the latter camp, warning that the window for intervention is narrowing. 'We have perhaps three to five years to lock in systems that are verifiably under human control,' he estimated. 'After that, the genie may not go back in the bottle.'
His speech concluded with a call for 'digital sovereignty' — the idea that nations must maintain independent oversight of AI rather than relying on corporate promises. 'Big tech wants us to trust their ethical commitments,' he said. 'But trust is not a protocol. We need verifiable, transparent mechanisms that guarantee human oversight at every level.'
As the sun set over the Thames, the audience of policymakers, engineers and journalists sat in unusual silence. The warning was clear: the future is not something that happens to us, but something we choose to build. And the choice, Amodei argued, must remain ours.
