In a stark warning delivered at the AI Safety Summit in London, Anthropic co-founder Dario Amodei declared that artificial intelligence systems must not be allowed to evolve autonomously without robust human supervision. Speaking to a packed auditorium of policymakers, technologists, and ethicists, Amodei argued that the rapid advancement of large language models and general-purpose AI tools is outpacing our ability to understand their long-term societal implications.
“We are building systems that can reason, generate, and act in ways we cannot fully predict,” Amodei said. “Without deliberate human oversight, we risk creating black box intelligences that could inadvertently cause harm on a global scale.” His remarks come days after Anthropic released its latest Claude model, which boasts improved reasoning capabilities but also triggered renewed debate over alignment and control.
Amodei emphasised that oversight is not about stifling innovation but about embedding safety protocols into the core architecture of AI systems. He called for a framework of “meaningful human involvement” at every stage of development, from training data curation to deployment decisions. This aligns with growing calls from ethicists who worry that fully autonomous AI could make unaccountable decisions in healthcare, finance, or criminal justice.
“The user experience of society is at stake here,” Amodei argued. “Every algorithm we deploy changes the digital ecosystem, and without human feedback loops, we are flying blind into a black mirror future.” His speech highlighted concrete proposals: mandatory impact assessments before releasing new models, transparency in training data sources, and the creation of independent auditing bodies.
The warning arrives amid a flurry of regulatory activity in both Europe and the United States. The EU’s AI Act is nearing final approval, while the White House recently secured voluntary commitments from major tech firms. However, critics argue that voluntary measures are insufficient. Amodei’s stance places Anthropic in the cautious camp, a position that has won praise from safety advocates but also drawn criticism from those who see it as a brake on innovation.
Quantum computing adds another layer of urgency. As these machines become more powerful, they could enable AI models to process data at speeds that make human oversight even more challenging. Amodei suggested that “digital sovereignty” will become a central issue, with nations needing to assert control over AI development to protect their citizens.
“We are not Luddites,” Amodei concluded. “We believe AI can solve humanity’s greatest challenges, from climate change to disease. But we must build it with caution, with humility, and with the human experience at its centre. The future is not something that happens to us; it is something we choose.”
His message resonated deeply with the audience, many of whom nodded as he painted a picture of a world where AI acts as a partner, not a master. Yet the clock is ticking. With each breakthrough, the gap between capability and control widens, making Amodei’s call for oversight not just prudent, but existential.
